End-to-end data management method and apparatus, and computer device and storage medium

Abstract

The present application relates to the field of data management. Disclosed are an end‑to‑end data management method and apparatus, and a computer device and a storage medium. The end‑to‑end data management method in the present application comprises: on the basis of an application scenario and service requirements, designing an automotive repair corpus classification template (S100); on the basis of the classification template, formulating a collection scheme, so as to acquire vehicle diagnostic and repair data; performing first preprocessing on the vehicle diagnostic and repair data to obtain normalized vehicle diagnostic and repair data (S300); then, performing reviewing to obtain standard vehicle diagnostic and repair data; and performing second preprocessing on the standard vehicle diagnostic and repair data to obtain data that meets requirements, and storing same in a database, so as to generate an automotive repair corpus. The end‑to‑end data management method in the present application covers data collection, cleaning, reviewing and storage, an end‑to‑end data management flow is constructed, the responsibilities of various stages are clearly defined, and the operational procedures are clear and well-organized, thereby ensuring efficient implementation and normalized execution of data management operations, and significantly improving the overall efficiency, quality and usability of data processing.

Description

A method, apparatus, computer device, and storage medium for end-to-end data management.

[0001] Cross-references to related applications

[0002] This application claims priority to Chinese Patent Application No. 2024119752348, filed on December 30, 2024, entitled "A Full-Link Data Management Method, Apparatus, Computer Equipment and Storage Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of data management technology, and in particular to a full-link data management method, apparatus, computer equipment, and storage medium. Background Technology

[0004] With the rapid development of artificial intelligence technology, especially the widespread application of large language models (such as GPT and BERT), data has become a core element in model development and application. During the training phase of large models, high-quality data directly impacts the model's accuracy, generalization ability, and practical performance. Similarly, during the application phase, data is crucial for achieving accurate reasoning and intelligent output. Therefore, data collection, development, management, and application permeate the entire lifecycle of large models, serving as key guarantees for improving model performance and ensuring successful application in various scenarios.

[0005] However, current data tools on the market mainly focus on loading local data into vector databases, typically providing basic data storage and retrieval capabilities, but with the following shortcomings:

[0006] Lack of data collection task management: Existing tools fail to cover functions such as task allocation, source management, and collection standard setting during the data collection phase, making it difficult to guarantee the efficiency and quality of data collection.

[0007] Lack of data development and cleaning functions: Most data tools lack standardized and normalized data processing mechanisms, making it impossible to efficiently preprocess, clean, and audit raw data, and thus difficult to generate high-quality data that can be used for training large models.

[0008] Insufficient corpus management and application: Current tools can usually only achieve simple data loading and storage, lacking flexible management of structured data and corpus classification templates, and cannot meet the needs of multiple application scenarios of data, such as data supply for intelligent retrieval and model inference interface calls.

[0009] Weak coordination and collaboration capabilities: Data tools lack unified management and process coordination throughout the entire data lifecycle, and cannot effectively coordinate the entire process from data collection to storage and application output.

[0010] Therefore, in response to the above problems, there is an urgent need to develop a full-chain data management method and tool to achieve unified management of data collection, development, cleaning, storage and application, connect all links of data from source to application, solve the shortcomings of data tools in task management, quality control, corpus classification management and application integration, provide high-quality data support for the research and development and practical application of large models, and further enhance their industry competitiveness and application effect.

[0011] Application content

[0012] In view of this, the embodiments of this application provide a full-link data management method, apparatus, computer equipment and storage medium, which can effectively solve the problems in the prior art that the data management process is imperfect, lacks a systematic collection, cleaning, review and application mechanism, resulting in low data quality, insufficient standardization and low efficiency in storage and external application.

[0013] In a first aspect, embodiments of this application provide a full-link data management method, including:

[0014] Design a classification template for automotive repair corpus based on application scenarios and business needs;

[0015] Based on the aforementioned automotive repair corpus classification template, a data collection plan was developed to obtain vehicle diagnostic and repair data.

[0016] The vehicle diagnostic and repair data is preprocessed for the first time to obtain standardized vehicle diagnostic and repair data; the standardized vehicle diagnostic and repair data is then reviewed to obtain standard vehicle diagnostic and repair data.

[0017] The standard vehicle diagnostic and repair data is preprocessed a second time to obtain qualified vehicle diagnostic and repair data to be stored in the database, thereby generating an automotive repair corpus.

[0018] In some embodiments, the step of designing an auto repair corpus classification template according to application scenarios and business needs includes: determining the target corpus content based on the application scenario;

[0019] Based on the content of the target corpus, the corpus is divided into categories, and corresponding classification criteria are formulated for each corpus category to form the auto repair corpus classification template.

[0020] In some embodiments, the step of formulating a data collection plan based on the automotive repair corpus classification template to obtain vehicle diagnostic and repair data includes:

[0021] Based on the automotive repair corpus classification template, the data collection sources for the vehicle diagnostics and repair data are analyzed and determined.

[0022] The data sources are evaluated, the scope and standards for data collection are determined, and data collection tasks are generated.

[0023] The data collection task is distributed to the corresponding devices for execution in order to obtain the vehicle diagnostic and maintenance data from different data sources.

[0024] In some embodiments, the first preprocessing of the vehicle diagnostic and repair data to obtain standardized vehicle diagnostic and repair data includes:

[0025] Based on a preset cleaning template, the vehicle diagnostic and repair data is formatted to obtain vehicle diagnostic and repair data with a uniform format.

[0026] The standardized vehicle diagnostic and repair data is input into a preset language model, and the standardized vehicle diagnostic and repair data is generated according to the format and content requirements of the automotive repair corpus classification template.

[0027] In some embodiments, the review of the standardized vehicle diagnostic and repair data to obtain standard vehicle diagnostic and repair data includes:

[0028] The accuracy and completeness of the standardized vehicle diagnostic and maintenance data are verified based on predetermined audit criteria.

[0029] The standardized vehicle diagnostic and repair data that does not meet the audit criteria are corrected or supplemented until they meet the audit criteria, and then the standard vehicle diagnostic and repair data is generated.

[0030] In some embodiments, the second preprocessing of the standard vehicle diagnostic and repair data to obtain compliant vehicle diagnostic and repair data to be stored and stored in a database to generate an automotive repair corpus includes:

[0031] The standard vehicle diagnostic and maintenance data is vectorized based on a vector model to obtain structured standard vehicle diagnostic and maintenance data.

[0032] According to predetermined storage rules, the structured standard vehicle diagnostic and repair data is stored in the database to obtain the automotive repair corpus.

[0033] In some embodiments, after performing a second preprocessing on the standard vehicle diagnostic and repair data to obtain compliant vehicle diagnostic and repair data to be stored and storing it in a database to generate an automotive repair corpus, the method further includes:

[0034] Based on the user's query request, the query data entered by the user is vectorized to generate a query vector;

[0035] Using a vector similarity algorithm, standard vehicle diagnostic and repair data with a high similarity to the query vector are retrieved from the automotive repair corpus;

[0036] Based on the search results, the standard vehicle diagnostic and repair data with the highest matching degree is determined, and the standard vehicle diagnostic and repair data is output through a preset interface.

[0037] Secondly, embodiments of this application provide a full-link data management device, comprising:

[0038] The template design module is used to design auto repair corpus classification templates according to application scenarios and business needs;

[0039] The data acquisition module is used to formulate a collection plan based on the automotive repair corpus classification template and acquire vehicle diagnostic and repair data;

[0040] The first preprocessing module is used to perform the first preprocessing on the vehicle diagnosis and repair data to obtain standardized vehicle diagnosis and repair data.

[0041] The data review module is used to review the standardized vehicle diagnostic and maintenance data to obtain standard vehicle diagnostic and maintenance data.

[0042] The automotive repair corpus generation module is used to perform a second preprocessing on the standard vehicle diagnostic and repair data to obtain qualified vehicle diagnostic and repair data to be stored in the database, thereby generating an automotive repair corpus.

[0043] Thirdly, embodiments of this application provide a computer device, the computer device including a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the end-to-end data management method of the first aspect described above.

[0044] Fourthly, embodiments of this application provide a computer-readable storage medium, wherein when the computer program is executed on a processor, it implements the full-link data management method of the first aspect described above.

[0045] The embodiments of this application have the following beneficial effects:

[0046] This application discloses a full-link data management method, apparatus, computer equipment, and storage medium. Based on application scenarios and business needs, it designs automotive repair corpus classification templates to achieve standardized data classification, formulates a collection plan to efficiently acquire vehicle diagnostic and repair data, and formats and standardizes the vehicle diagnostic and repair data through a first preprocessing step, resolving issues such as inconsistent and redundant raw data formats. The standardized vehicle diagnostic and repair data is then reviewed to ensure its accuracy and completeness, resulting in standard vehicle diagnostic and repair data. A second preprocessing step vectorizes and structures the standard vehicle diagnostic and repair data, ensuring it meets storage and retrieval requirements, and stores it in a database, ultimately generating a high-quality automotive repair corpus. This solution addresses the entire data management chain, including collection, cleaning, review, and storage, achieving full-link data management. The method effectively solves problems such as lack of data collection task management, insufficient data standardization, and inconvenience in external application of the corpus, improving data quality and usability. Attached Figure Description

[0047] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 illustrates an application scenario of a full-link data management method according to an embodiment of this application.

[0049] Figure 2 shows a flowchart of a full-link data management method according to an embodiment of this application;

[0050] Figure 3 shows a schematic diagram of the structure of a full-link data management device according to an embodiment of this application. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0052] In the following text, the terms "comprising," "having," and their cognates, which may be used in various embodiments of this application, are intended only to indicate a particular feature, number, step, operation, element, component, or combination thereof, and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations thereof, or adding the possibility of one or more combinations thereof. Furthermore, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0053] Unless otherwise specified, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of this application pertain. Terms (such as those defined in commonly used dictionaries) shall be interpreted as having the same meaning as in their contextual meaning in the relevant technical field and shall not be construed as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of this application.

[0054] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0055] Considering the shortcomings of existing data management processes and the lack of systematic collection, cleaning, review, and application mechanisms, resulting in low data quality, insufficient standardization, and inefficient storage and external application, this paper proposes a full-link data management method. This method involves designing automotive repair corpus classification templates, formulating collection plans, performing data cleaning and preprocessing on vehicle diagnostic and repair data, reviewing the quality of standardized vehicle diagnostic and repair data, obtaining compliant vehicle diagnostic and repair data for storage, and storing the compliant data in a database. Ultimately, this achieves efficient data management and external application, realizing full-link data management encompassing collection, cleaning, review, and storage tasks. Furthermore, through task dispatch processes and automated workflows, clear division of responsibilities and efficient collaborative operation processes are ensured at each stage, further improving the execution efficiency and accuracy of data management, and optimizing data quality and usability.

[0056] This application provides a full-link data management method applicable to the application environment shown in Figure 1. Specifically, the full-link data management method is applied in a computer system, which includes a client and a server as shown in Figure 1. The client and server communicate via a network. The client, also known as a user terminal, refers to a program that provides local services to the client in relation to the server. The client can be installed on, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable diagnostic devices (e.g., diagnostic boxes). The server can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0057] Figure 2 shows a flowchart of a full-link data management method according to an embodiment of this application. Exemplarily, the full-link data management method includes the following steps:

[0058] Step S100: Design a classification template for automotive repair corpus based on application scenarios and business needs.

[0059] For example, application scenarios refer to the specific usage environments and functional requirements that the automotive repair corpus data is intended to serve, such as, but not limited to, fault diagnosis scenarios, repair guidance scenarios, circuit testing scenarios, and parts query scenarios. Business requirements are specific data requirements proposed based on the application scenarios. For example, they include, but are not limited to, providing a complete corpus containing core content such as fault code definitions, fault code symptoms, repair guidance, circuit inspection, circuit testing, component testing, and repair guides. For example, the corpus needs to be accurate, complete, and standardized to meet the requirements of repair guidance and diagnostic systems; the data needs to have a unified structure and format to facilitate quick system access and retrieval. Automotive repair classification templates refer to the data classification and content structure standards designed according to application scenarios and business requirements, used to guide data collection, management, and application. Examples include fault code corpus templates, circuit diagram corpus templates, and parts description corpus templates.

[0060] In one embodiment, in step S100, a classification template for automotive repair corpus is designed based on the application scenario and business requirements, including:

[0061] The target corpus content is determined based on the application scenario described.

[0062] For example, application scenarios can include: Fault diagnosis scenario: When a user enters a fault code, the system needs to return an explanation of the fault code and a repair solution to help repair personnel quickly locate and troubleshoot the problem. Repair guidance scenario: Providing standardized repair manual content for repair engineers or intelligent repair assistants, including circuit testing, component testing, and repair procedures. Parts search scenario: Providing parts descriptions, parameters, models, and replacement guidance to ensure accurate parts selection during the repair process. Based on the target corpus content, the corpus is divided into categories, and corresponding classification standards are developed for each corpus category to form an automotive repair corpus classification template.

[0063] After identifying the application scenario, determine the core information that needs to be provided within that scenario, i.e., the target corpus content. For example: Fault code scenario: fault code definition, fault symptoms, circuit testing, component testing, repair guide, etc.; Circuit testing scenario: circuit diagram, signal testing, fault location methods, etc.; Component testing scenario: component name, specifications, testing standards, and replacement steps; Parts scenario: part name, part number, compatible vehicle models, parameter specifications, etc.

[0064] After determining the target corpus content, the data needs to be scientifically classified and organized into different corpus categories. Each category is divided based on the corpus's usage function and content attributes. For example:

[0065] Fault code corpus: Definitions, symptoms, detection methods, and repair guidelines for fault codes and their associated information.

[0066] Circuit diagram corpus: circuit structure diagrams, test points, and signal detection methods.

[0067] Component corpus: Specifications, testing methods, installation and replacement procedures for components.

[0068] Repair manual corpus: standardized repair steps and procedures.

[0069] Furthermore, classification criteria provide a unified definition of the content structure, field requirements, and format specifications for each corpus category. This ensures the integrity, consistency, and usability of the data. For example, classification criteria include: Field definitions: the specific fields that each corpus category must include and their meanings; Format specifications: the format of data storage (e.g., JSON, XML, or tabular structure); Content specifications: the requirements for filling in data content, such as required fields, character length, and terminology consistency.

[0070] After defining the target content requirements, corpus categories, and classification standards, a final automotive repair corpus classification template is created. For example, the corpus classification template for fault codes includes: fault code, brand, vehicle model, model year, circuit inspection, circuit testing, component testing, and repair guide. The automotive repair corpus classification template is a standardized data framework designed for vehicle diagnostic and repair data, with a clear classification structure and field requirements, providing a unified standard for data collection, cleaning, review, and storage.

[0071] Step S200: Develop a data collection plan based on the automotive repair corpus classification template to obtain vehicle diagnostic and repair data.

[0072] For example, a data acquisition plan refers to the development of a plan and execution process for acquiring vehicle diagnostic and repair data based on the content standards and data requirements defined in the automotive repair corpus classification template. For example, a data acquisition plan includes the following key elements: (1) Data sources: public data, data from cooperative channels, offline data, purchased data, etc. (2) Scope of data acquisition: fault code definitions, fault phenomena, circuit diagrams, repair procedures, etc. Specify the coverage of the data, including dimensions such as brand, model, year, and system components. (3) Data acquisition standards: Define the format requirements of the data and formulate the quality standards of the data, such as: data fields must be complete, data content must be accurate, and unambiguous. (4) Task execution process: Decompose the data acquisition task into multiple sub-tasks and assign them to the data acquisition equipment. Set the time nodes and priorities for task completion to ensure the orderly progress of the data acquisition work.

[0073] Data collection is performed by personnel or automated tools according to the data sources and standards defined in the data collection plan. For example, this includes downloading vehicle repair manuals and fault code tables from online databases, obtaining fault diagnosis records for a specific brand of vehicle through partner channels, and scanning, digitizing, and organizing offline paper documents into structured data.

[0074] The final vehicle diagnostic and repair data obtained includes: Fault code information: fault code number, definition, and fault symptoms. Circuit testing methods: circuit diagram, test procedures, and parameter standards. Component testing information: component testing standards, testing methods, and replacement instructions. Repair guidance information: troubleshooting steps and solutions from the repair manual.

[0075] In one embodiment, step S200 involves developing a data collection plan based on an automotive repair corpus classification template to acquire vehicle diagnostic and repair data, including:

[0076] Based on the classification template of the automotive repair corpus, the sources of vehicle diagnostic and repair data were analyzed and determined.

[0077] As an example, guided by a well-designed automotive repair corpus classification template, the first step is to analyze and determine the data collection sources, i.e., the channels for obtaining vehicle diagnostic and repair data. Data collection sources can be divided into the following categories: (1) Public data: Repair materials publicly available online, such as technical documents from car manufacturers' official websites, online repair manuals, industry standard databases, etc. (2) Collaborative channel data: Cooperating with car manufacturers, repair shops, and third-party data providers to obtain high-quality vehicle diagnostic and repair data. (3) Offline data: Manuals and paper repair records provided by repair shops and technicians, which need to be digitized. (4) Purchased data: Purchasing vehicle fault diagnosis and repair datasets through professional data providers.

[0078] For example, in the "vehicle fault code diagnosis" scenario, the data sources can include: technical manuals published on the official website of the car manufacturer, fault code diagnosis records obtained in cooperation with repair shops, and historical fault repair data provided by data providers.

[0079] The data sources are assessed, the scope and standards for data collection are defined, and data collection tasks are generated.

[0080] In detail, reliable data sources are selected by conducting feasibility and quality assessments of the identified data sources. Assessment indicators include: Data accuracy: Is the source credible and the data content authoritative? Data completeness: Can the source provide all the necessary information (e.g., fault codes, testing methods, repair guides, etc.)? Data accessibility: Is the source easily accessible, for example, does it require authorization, cooperation agreements, or payment? Data timeliness: Is the source data updated promptly and does it meet current vehicle models and technical requirements?

[0081] Secondly, clearly define the scope of data to be collected, including: brand and vehicle model, production year, and data categories: fault code data, circuit test data, component test data, repair manual data, etc. Then, establish collection standards to regulate the format and quality of the collected data, including: Data format: standardized to JSON, XML, or CSV formats for easy subsequent processing; Content specifications: ensuring field completeness, for example, fault codes must include "definition, symptoms, and testing steps"; Data quality requirements: ensuring data is free of redundancy and errors, and meets accuracy and usability standards.

[0082] Finally, based on the data source, scope, and standards, specific data collection tasks are generated, including: Task objective: Which data to collect. Data source: Specify the data source. Data standards: Format, field content, and quality requirements. Execution method: Whether the task is automated or requires manual intervention.

[0083] Data collection tasks are distributed to appropriate devices for execution to acquire vehicle diagnostic and maintenance data from various sources. Exemplarily, data collection tasks are distributed to suitable acquisition devices or tools to perform data acquisition. Task distribution methods include: Automated tools: such as web crawlers or API call scripts, used to automatically collect data from online databases or partner channels. Manual operation: Data acquisition personnel manually obtain data from offline documents and maintenance records and enter it into the system. Hybrid approach: Some data is automatically collected, while others require manual verification and supplementation. Through task execution, vehicle diagnostic and maintenance data that meets the template requirements is obtained from different sources, supporting subsequent data processing and storage.

[0084] Step S300: Perform the first preprocessing on the vehicle diagnostic and repair data to obtain standardized vehicle diagnostic and repair data.

[0085] As an example, the first preprocessing mainly addresses the format, content, and quality issues in vehicle diagnostic and repair data, making the data standardized and consistent, laying a foundation for subsequent review, storage, and application. For example, the following operations are performed on the collected vehicle diagnostic and repair data: (1) Data formatting: Convert the vehicle diagnostic and repair data into a unified standard format (e.g., JSON, XML, CSV, etc.). Address the issue of inconsistent data formats from different sources. (2) Data content standardization: Clean and organize the data content, removing redundant information, irrelevant content, or erroneous data. Standardize field naming, terminology standards, and unit representations to ensure that the data content is standardized and consistent. (3) Data filtering and deduplication: Deduplicate the collected data and delete duplicate content. Filter out data that meets the requirements of the automotive repair corpus classification template and remove irrelevant or non-compliant parts. (4) Field completion and verification: Complete missing fields in the data or mark them as pending processing. Verify the completeness and preliminary accuracy of the data to ensure that all necessary fields have values.

[0086] After the first preprocessing, the generated standardized vehicle diagnostic and repair data has the following characteristics:

[0087] Uniform format: All data uses a consistent data format, such as JSON or XML.

[0088] Content standardization: Field names, terminology standards, and unit representations have all been standardized, and the data content is clear and unambiguous.

[0089] Improved integrity: Missing parts of the data have been filled in or marked, ensuring data integrity and availability.

[0090] Redundant data cleanup: Duplicate and invalid data are removed, making the data more concise.

[0091] In one embodiment, step S300 involves a first preprocessing of the vehicle diagnostic and repair data to obtain standardized vehicle diagnostic and repair data, including:

[0092] Based on the preset cleaning template, the vehicle diagnostic and repair data is formatted to obtain vehicle diagnostic and repair data with a uniform format.

[0093] For example, a data cleaning template is a pre-defined data processing specification and standard rule used to guide the formatting and content standardization of data. Cleaning templates are typically designed by business experts and model experts based on application requirements and corpus templates, and include: data field requirements (e.g., fault codes, fault definitions, maintenance procedures, etc.), data format standards (e.g., JSON, XML, CSV), and content specification requirements (e.g., terminology standardization, unit unification, mandatory field settings, etc.).

[0094] Standardizing the collected vehicle diagnostic and repair data according to the cleaning template resolves issues such as inconsistent data formats, incomplete or redundant fields, and inconsistencies in terminology and units. Then, the standardized vehicle diagnostic and repair data is input into a pre-defined language model, which generates standardized vehicle diagnostic and repair data based on the format and content requirements of the automotive repair corpus classification template.

[0095] Input standardized vehicle diagnostic and repair data into a pre-defined language model, and generate standardized vehicle diagnostic and repair data according to the format and content requirements of the automotive repair corpus classification template.

[0096] For example, a language model refers to a pre-trained large-scale natural language processing model (e.g., GPT, BERT), whose function is to automatically generate or supplement structured and standardized data content based on input data and preset rules. The automotive repair corpus classification template defines the format and content requirements for vehicle diagnostic and repair data, serving as a guiding standard for the language model's output. The formatted data is passed as input to the language model. Based on the requirements of the automotive repair corpus classification template, the language model automatically completes, standardizes, and organizes the data content, generating standardized vehicle diagnostic and repair data that conforms to the standards.

[0097] For example, the input parameters can be fault codes and repair manuals, while the language model outputs the following parameters based on the format and content requirements of the auto repair classification template: fault code, brand, model, year, circuit inspection, circuit test, component test, and repair guide.

[0098] Furthermore, after formatting and automated generation of language models, the resulting standardized vehicle diagnostic and maintenance data has the following characteristics:

[0099] Uniform format: All data meets the template's format requirements and is suitable for system storage and retrieval.

[0100] Content complete: missing fields (e.g., fault symptoms, repair guides, etc.) are automatically completed and standardized.

[0101] Standardized output: Conforms to the requirements of the automotive repair corpus classification template, with standardized and accurate content.

[0102] High applicability: It provides high-quality data support for subsequent auditing, storage and application (e.g., intelligent diagnostics, maintenance guidance systems).

[0103] Step S400: Review the standardized vehicle diagnostic and maintenance data to obtain standard vehicle diagnostic and maintenance data.

[0104] For example, the specific operations for auditing standardized vehicle diagnostic and maintenance data include the following aspects: (1) Field integrity check: Ensure that each piece of data contains all the required fields required by the template, such as: fault code, fault definition, fault symptoms, circuit testing method, component testing standard, and maintenance guide; (2) Content accuracy verification: Compare whether the data content conforms to the actual business standards. For example, whether the definition of the fault code is consistent with the information in the industry standard or maintenance manual. Whether the fault symptoms, testing steps and maintenance guide have correct logic and operation guidance. For example, in the original data, the fault definition of "P0123" is "circuit fault". The audit found that the content was too general and needed to be corrected to a specific definition, etc. (3) Data consistency verification: Unify terminology, format and unit: Ensure that all data uses standard terminology and that the unit and format are consistent. (4) Data compliance check: Ensure that the data source is reliable and conforms to business specifications, especially third-party data or manually entered data, which needs to be verified twice. (5) Audit feedback and correction: Correct or supplement the data problems found during the audit. If some data cannot be corrected, it needs to be marked as abnormal and not included in the standard dataset, etc.

[0105] In one embodiment, step S400 involves reviewing the standardized vehicle diagnostic and repair data to obtain standard vehicle diagnostic and repair data, including:

[0106] Based on predetermined audit standards, the accuracy and completeness of standardized vehicle diagnostic and repair data are verified; standardized vehicle diagnostic and repair data that does not meet the audit standards are corrected or supplemented until the standardized vehicle diagnostic and repair data meets the audit standards, and standard vehicle diagnostic and repair data is generated.

[0107] The audit criteria are pre-defined data quality assessment rules used to verify standardized data and ensure that the data meets quality requirements. Standardized vehicle diagnostic and maintenance data are verified item by item through automated verification tools or manual checks to check whether they meet the audit criteria. If problems are found during the audit process, the following actions can be taken: (1) Correcting errors: Correcting inaccurate or non-standard field content. For example: replacing incorrect fault code definitions with correct standard definitions. (2) Supplementing missing content: Supplementing and improving missing fields or content. For example: supplementing the specific description of the "fault symptoms" field, such as "unstable idling speed, weak acceleration". (3) Re-verifying: Re-verifying the corrected or supplemented data to ensure that it meets the audit criteria. (4) Iterative correction process: If the corrected or supplemented data still does not fully meet the standards, the correction process is repeated until the standardized vehicle diagnostic and maintenance data passes the audit and standard vehicle diagnostic and maintenance data is generated.

[0108] Step S500: Perform a second preprocessing on the standard vehicle diagnostic and repair data to obtain the required vehicle diagnostic and repair data to be stored and store it in the database to generate an automotive repair corpus.

[0109] As an example, the second preprocessing involves in-depth processing and structural optimization of standard vehicle diagnostic and maintenance data, such as vectorization, data compression, deduplication, field supplementation, and format standardization, to make the data more standardized and structured, meet the design requirements of database storage, and generate corresponding vector representations to improve data retrieval and consistency.

[0110] Subsequently, the data that has undergone a second preprocessing step and meets the requirements is imported into the database according to predetermined storage rules. These storage rules may include field structure definitions, index creation, data format standardization, and the selection of storage methods; for example, relational databases are used to store field data, and vector databases are used to store vector data.

[0111] Ultimately, this data is stored in a database to form an automotive repair corpus, a standardized and structured collection of vehicle diagnostic and repair data. The automotive repair corpus not only supports rapid data retrieval but also provides high-quality data support for intelligent diagnostic systems and fault analysis platforms, meeting the practical application needs of vehicle repair and troubleshooting.

[0112] In one embodiment, in step S500, the standard vehicle diagnostic and repair data is preprocessed a second time to obtain the required vehicle diagnostic and repair data to be stored and stored in the database to generate an automotive repair corpus. This includes: vectorizing the standard vehicle diagnostic and repair data based on a vector model to obtain structured standard vehicle diagnostic and repair data.

[0113] As an example, vector models are a commonly used model in deep learning to transform unstructured data into computable vector representations. In data processing, common tabular data (e.g., fields in a database) is called structured data. This type of data has a fixed format, making it easy to process and store. Unstructured data (e.g., text, images, audio, video, etc.), due to its complex form, cannot be directly processed using traditional structured methods. Therefore, it is necessary to transform this data into a form suitable for computation and analysis; this process is called vectorization.

[0114] Vectorization is the process of transforming unstructured data into multidimensional floating-point arrays, which can be used to represent the semantics and features of the data. For example, a vectorized array might be in the form of [0.3333, 0.1224, 0.3222, ...], where each value represents a measure of the semantic features of the data in the corresponding dimension. The computational model used for vectorization is called a vector model, also known as an embedding model, which uses deep learning algorithms to extract semantic information from the data and generate a high-dimensional vector representation.

[0115] During vectorization, standard vehicle diagnostic and repair data (e.g., fault codes, fault definitions, repair guides, etc., in text, image, audio, and video formats) are typically vectorized in a block-based manner. For example, an article can be divided into multiple blocks by paragraphs, and each block can be vectorized independently to generate multiple sets of corresponding floating-point vectors. This processing method preserves the semantic features of the data while facilitating subsequent storage, retrieval, and analysis.

[0116] According to predetermined storage rules, structured standard vehicle diagnostic and repair data are stored in the database to obtain an automotive repair corpus.

[0117] As an example, standard vehicle diagnostic and repair data, after undergoing vectorization and structuring processing, is stored in the database. The data is converted into multi-dimensional floating-point numbers (vector representation), a form that highly abstracts the semantic features of the data. Unstructured content such as fault codes, fault definitions, and repair guides are stored as high-dimensional floating-point vectors, unlike the text content stored in traditional databases. Once stored in the database, this forms an automotive repair corpus containing standardized and structured vehicle diagnostic and repair data, supporting semantic retrieval and intelligent applications.

[0118] It's important to note that the content stored in the database is not traditional visible text or fields, but rather high-dimensional vector data. Fault code definitions and their corresponding content are converted to floating-point numbers and stored in the database. Therefore, the data in the database is not visible; it exists only as a massive amount of floating-point numbers. This vector data does not directly correspond to the original field content, but rather represents a multi-dimensional abstraction of its semantic features.

[0119] In one embodiment, after performing a second preprocessing on standard vehicle diagnostic and repair data to obtain compliant vehicle diagnostic and repair data to be stored in a database to generate an automotive repair corpus, the process further includes:

[0120] Based on the user's query request, the query data entered by the user is vectorized to generate a query vector.

[0121] As an example, when a user initiates a query request, the user-input query data, such as fault descriptions, keywords, and fault codes, is vectorized. This means that the query content is converted into a set of numerical vectors—a multidimensional array of floating-point numbers—using a pre-defined vector model. This vector representation captures the semantic information of the query content, enabling the system to understand its deeper meaning.

[0122] Using vector similarity algorithms, standard vehicle diagnostic and repair data with high vector similarity are retrieved and queried from the automotive repair corpus.

[0123] As an example, since the standard vehicle diagnostic and repair data stored in the automotive repair corpus has been vectorized, each data block corresponds to a high-dimensional floating-point vector. Therefore, using vector similarity algorithms (e.g., cosine similarity or Euclidean distance), the similarity between the query vector and each stored vector in the automotive repair corpus can be calculated, and the few data blocks with the highest similarity can be retrieved.

[0124] Based on the search results, the standard vehicle diagnostic and repair data with the highest matching degree is determined, and the standard vehicle diagnostic and repair data is output through a preset interface.

[0125] Specifically, by sorting the retrieved block vectors by similarity, the vector blocks with the highest matching degree are determined. These vector blocks correspond to the original text content in the automotive repair corpus. The vector blocks can be mapped back to readable vehicle diagnosis and repair data. Then, a large model (e.g., a deep learning model, an AI model, etc.) is called to organize and process the vehicle diagnosis and repair data. Finally, the generated more logical, coherent and professional output results are returned or displayed to the user through the API interface (i.e., application programming interface).

[0126] It is worth noting that throughout the data management process, this application achieved efficient and accurate data management through automated task assignment and workflow mechanisms. By setting completion times for each workflow and tracking progress in real time, the orderly connection between workflows was ensured, significantly improving the execution efficiency of the data management chain.

[0127] This application's end-to-end data management method, by combining application scenarios and business needs, designs an automotive repair corpus classification template to achieve standardized data classification and formulates a collection plan to obtain vehicle diagnostic and repair data. The collected data undergoes a first preprocessing stage to format and standardize the data, resolving issues such as inconsistent raw data formats and redundancy. The accuracy and completeness of the data are verified through an auditing process to obtain standard vehicle diagnostic and repair data. Further, a second preprocessing stage vectorizes and structures the standard data to meet storage and retrieval requirements, ultimately storing it in a database to build a high-quality automotive repair corpus. This solution revolves around the entire data management process, covering data collection, cleaning, auditing, and storage. The method of this application achieves closed-loop data management, significantly improving data quality and usability.

[0128] This application also proposes a full-link data management device, as shown in Figure 3, which includes:

[0129] Template design module 31 is used to design auto repair corpus classification templates according to application scenarios and business needs;

[0130] Data acquisition module 32 is used to formulate a collection plan based on the automotive repair corpus classification template to acquire vehicle diagnostic and repair data;

[0131] The first preprocessing module 33 is used to perform the first preprocessing of vehicle diagnostic and repair data to obtain standardized vehicle diagnostic and repair data.

[0132] The data review module 34 is used to review standardized vehicle diagnostic and maintenance data to obtain standard vehicle diagnostic and maintenance data.

[0133] The automotive repair corpus generation module 35 is used to perform a second preprocessing on standard vehicle diagnostic and repair data to obtain qualified vehicle diagnostic and repair data to be stored in the database, thereby generating an automotive repair corpus.

[0134] It is understood that the apparatus of this embodiment corresponds to the method of the above embodiments, and the options in the above embodiments are also applicable to this embodiment, so they will not be described again here.

[0135] This application also provides a computer device, exemplary of which includes a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to enable the device to perform the functions of the various modules in the above-described end-to-end data management method or end-to-end data management device.

[0136] The processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including at least one of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Network Processor (NP), Digital Signal Processor (DSP), Application-Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0137] The memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory is used to store computer programs, and the processor can execute the computer programs accordingly after receiving execution instructions.

[0138] This application also provides a computer-readable storage medium for storing the computer program used in the aforementioned computer device. For example, the computer-readable storage medium may include, but is not limited to, various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0139] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that, in alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0140] In addition, the functional modules or units in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0141] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a smartphone, personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0142] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for end-to-end data management, characterized in that, The method includes: Design a classification template for automotive repair corpus based on application scenarios and business needs; Based on the aforementioned automotive repair corpus classification template, a data collection plan was developed to obtain vehicle diagnostic and repair data. The vehicle diagnostic and repair data is preprocessed for the first time to obtain standardized vehicle diagnostic and repair data; the standardized vehicle diagnostic and repair data is then reviewed to obtain standard vehicle diagnostic and repair data. The standard vehicle diagnostic and repair data is preprocessed a second time to obtain qualified vehicle diagnostic and repair data to be stored in the database, thereby generating an automotive repair corpus.

2. The end-to-end data management method according to claim 1, characterized in that, The above describes the design of a corpus classification template for auto repair based on application scenarios and business needs, including: Determine the target corpus content based on the application scenario described above; Based on the content of the target corpus, the corpus is divided into categories, and corresponding classification criteria are formulated for each corpus category to form the auto repair corpus classification template.

3. The end-to-end data management method according to claim 1, characterized in that, The process of developing a data collection scheme based on the automotive repair corpus classification template to obtain vehicle diagnostic and repair data includes: Based on the automotive repair corpus classification template, the data collection sources for the vehicle diagnostics and repair data are analyzed and determined. The data sources are evaluated, the scope and standards for data collection are determined, and data collection tasks are generated. The data collection task is distributed to the corresponding devices for execution in order to obtain the vehicle diagnostic and maintenance data from different data sources.

4. The end-to-end data management method according to claim 1, characterized in that, The first preprocessing of the vehicle diagnostic and repair data to obtain standardized vehicle diagnostic and repair data includes: Based on a preset cleaning template, the vehicle diagnostic and repair data is formatted to obtain vehicle diagnostic and repair data with a uniform format. The standardized vehicle diagnostic and repair data is input into a preset language model, and the standardized vehicle diagnostic and repair data is generated according to the format and content requirements of the automotive repair corpus classification template.

5. The end-to-end data management method according to claim 1, characterized in that, The process of reviewing the standardized vehicle diagnostic and repair data to obtain standard vehicle diagnostic and repair data includes: The accuracy and completeness of the standardized vehicle diagnostic and maintenance data are verified based on predetermined audit criteria. The standardized vehicle diagnostic and repair data that does not meet the audit criteria are corrected or supplemented until they meet the audit criteria, and then the standard vehicle diagnostic and repair data is generated.

6. The end-to-end data management method according to claim 1, characterized in that, The standard vehicle diagnostic and repair data is preprocessed a second time to obtain qualified vehicle diagnostic and repair data to be stored in the database, thereby generating an automotive repair corpus, including: The standard vehicle diagnostic and maintenance data is vectorized based on a vector model to obtain structured standard vehicle diagnostic and maintenance data. According to predetermined storage rules, the structured standard vehicle diagnostic and repair data is stored in the database to obtain the automotive repair corpus.

7. The end-to-end data management method according to claim 1, characterized in that, After performing a second preprocessing on the standard vehicle diagnostic and repair data to obtain compliant vehicle diagnostic and repair data to be stored in the database to generate an automotive repair corpus, the process further includes: Based on the user's query request, the query data entered by the user is vectorized to generate a query vector; Using a vector similarity algorithm, standard vehicle diagnostic and repair data with a high similarity to the query vector are retrieved from the automotive repair corpus; Based on the search results, the standard vehicle diagnostic and repair data with the highest matching degree is determined, and the standard vehicle diagnostic and repair data is output through a preset interface.

8. A full-link data management device, characterized in that, The device includes: The template design module is used to design auto repair corpus classification templates according to application scenarios and business needs; The data acquisition module is used to formulate a collection plan based on the automotive repair corpus classification template and acquire vehicle diagnostic and repair data; The first preprocessing module is used to perform the first preprocessing on the vehicle diagnosis and repair data to obtain standardized vehicle diagnosis and repair data. The data review module is used to review the standardized vehicle diagnostic and maintenance data to obtain standard vehicle diagnostic and maintenance data. The automotive repair corpus generation module is used to perform a second preprocessing on the standard vehicle diagnostic and repair data to obtain qualified vehicle diagnostic and repair data to be stored in the database, thereby generating an automotive repair corpus.

9. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the end-to-end data management method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed on a processor, implements the end-to-end data management method according to any one of claims 1-7.

Images