Data analysis method and operation and maintenance system

Abstract

A data analysis method and an operation and maintenance system applied to the field of intelligent driving, for use in combining a large model and vehicle multi-modal data to provide more accurate problem feedback for a user. The method comprises: acquiring input data, the input data being obtained on the basis of data inputted by the user, and the input data indicating a vehicle problem to be solved; using the input data as an input of a large language model, and outputting a problem category, that is, using the large language model to classify the vehicle problem; then acquiring multi-modal data, the multi-modal data comprising data collected by at least one sensor in a vehicle, and the multi-modal data comprising data related to the vehicle problem, such as data collected by a sensor in a vehicle when a vehicle problem occurs; then performing analysis on the basis of the problem category and the multi-modal data to obtain a problem analysis result; and then outputting the problem analysis result.

Description

A data analysis method and operation and maintenance system Technical Field

[0001] This application relates to the field of intelligent driving, and in particular to a data analysis method and operation and maintenance system. Background Technology

[0002] As more and more vehicles are equipped with intelligent driving systems, problems during use are gradually emerging, requiring more effective ways to address these issues. Voice feedback is one of the important means of interaction between users and the system; however, the content of voice feedback is often ambiguous, the volume of feedback is enormous, and there is a lack of effective prioritization.

[0003] For example, some existing solutions pre-configure audio settings, allowing users to select a suitable voice response when they provide feedback via voice. However, pre-configured audio is inflexible and may fail to provide appropriate answers, resulting in a poor user experience. Summary of the Invention

[0004] This application provides a data analysis method and operation and maintenance system for combining large models and vehicle multimodal data to provide users with more accurate problem feedback.

[0005] In view of this, firstly, this application provides a data analysis method, comprising: acquiring input data, the input data being obtained based on user input data, the input data indicating the solution of a vehicle problem; using the input data as input to a large language model, outputting a problem category, i.e., classifying the vehicle problem using the large language model; subsequently acquiring multimodal data, the multimodal data including data collected by at least one sensor in the vehicle, the multimodal data including data related to the vehicle problem, such as data collected by sensors in the vehicle when the vehicle problem occurs; subsequently analyzing the problem category and the multimodal data to obtain a problem analysis result; and subsequently outputting the problem analysis result.

[0006] In this embodiment of the application, after obtaining user input data, the recognition capability of the large language model can be used to identify problems more flexibly. Then, problem analysis is performed based on the problem category and multimodal data related to the problem. In this way, the large language model and multimodal data are combined to more accurately analyze the problems that users need to provide feedback or suggestions on, and output more accurate problem feedback in a targeted manner.

[0007] In one possible implementation, the aforementioned process of using input data as input to a large language model and outputting a question category includes: using input data as input to the large language model and outputting a question category and emotional information, whereby the emotional information includes the user's emotion corresponding to the input data, and the input data includes voice data; the aforementioned process of outputting the question analysis result includes: outputting the question analysis result and the emotional information. In this implementation, the emotion recognition capability of the large language model can also be utilized to output the emotional information corresponding to the input data, such as the user's emotion category or emotion score. This allows maintenance personnel to address issues requiring manual processing based on the emotional information, or to provide targeted output based on the user's emotion when outputting the question analysis result to the user, thereby improving the user experience.

[0008] In one possible implementation, the aforementioned large language model is used to filter question categories from multiple categories in a pattern library. In this embodiment, a pattern library containing multiple question categories can be pre-set, enabling the large language model to identify the category corresponding to the question requiring user feedback or suggestions based on this pattern library.

[0009] In one possible implementation, the aforementioned pattern library further includes data rules corresponding to each category, where each data rule represents a data type corresponding to that category. The aforementioned acquisition of multimodal data may include: determining a first data rule corresponding to a problem category based on the pattern library; this first data rule may include, for example, the type of data required for analyzing the problem corresponding to that category; and acquiring multimodal data based on the first data rule. This multimodal data includes data of the type indicated by the first data rule. This multimodal data may originate from a vehicle. When the method provided in this embodiment is deployed in the cloud, the cloud can receive multimodal data uploaded actively by the vehicle or requested by the cloud. When the method provided in this embodiment is deployed on a vehicle, the vehicle can directly read the multimodal data. Therefore, in this embodiment, multimodal data can be acquired based on the data rules set in the pattern library, thereby obtaining the multimodal data required for analysis.

[0010] In one possible implementation, the aforementioned pattern library also includes model types corresponding to each category, that is, the types of models that can be used when analyzing problems of each category. The aforementioned analysis based on problem categories and multimodal data to obtain problem analysis results may specifically include: determining the model type corresponding to the problem category based on the pattern library, referred to as the first model type for ease of distinction; and then analyzing the multimodal data based on the model indicated by the first model type to obtain problem analysis results. Therefore, in this embodiment, problem analysis can be performed according to pre-deployed models corresponding to problem categories, allowing for more targeted analysis of each category of problems to obtain more accurate problem analysis results.

[0011] In one possible implementation, the aforementioned output of the problem analysis results includes: inputting the problem analysis results into a question-and-answer system, outputting a question response, wherein the question system is a system for responding to users and can be used to convert the problem analysis results into interactive content that is easy for users to understand. The question response includes the content of the problem analysis results, such as language or wording that users can understand, and is used to answer vehicle-related problems corresponding to the input data; subsequently, the question response is output through an interactive interface, which can be an interactive interface on the vehicle or an interactive interface on the user's terminal. In this embodiment of the application, a question-and-answer system can be used to output more easily understood answers to users and display them to users in an interactive interface to answer users' questions or suggestions, thereby improving the user experience.

[0012] In one possible implementation, the aforementioned output of the problem analysis results may further include: generating a problem description based on the problem analysis results. The problem description includes the problem analysis results and an analysis path, whereby the analysis path indicates the cause of the problem indicated by the input data. Optionally, if the large language model also outputs sentiment information, the problem description may also include the sentiment information. In this embodiment, the problem analysis results and the problem analysis path may also be output, whereby the analysis path indicates the root cause of the problem, enabling maintenance personnel to perform more accurate analysis of the problem or suggestion, thus facilitating problem resolution. Furthermore, the user's sentiment information may also be output, which allows maintenance personnel to prioritize problem resolution, such as prioritizing problems with higher user sentiment scores to improve user experience.

[0013] In one possible implementation, the aforementioned method further includes: acquiring historical problem information, which includes information related to the problems corresponding to the input data; and analyzing the historical problem information based on the problem category and multimodal data to obtain problem analysis results. In this embodiment, historical problems can also be referenced to analyze the current problem, and the accuracy and efficiency of the analysis can be improved by combining the results of historical problems or feedback information from those historical problems.

[0014] Secondly, this application discloses an operation and maintenance system, comprising: a vehicle-side terminal and a cloud-based terminal;

[0015] In the cloud, input data is used as input to a large language model, and the output is the problem category. The input data indicates how to solve the problem on the vehicle side.

[0016] The cloud is also used to acquire multimodal data, which includes data collected by at least one sensor and data related to vehicle issues.

[0017] In the cloud, it is also used to analyze problems based on problem categories and multimodal data to obtain problem analysis results;

[0018] In the cloud, it is also used to output problem analysis results.

[0019] The effects achieved by the second aspect or any optional implementation of the second aspect of this application can be referred to the description of the first aspect or any optional implementation of the first aspect above, and will not be repeated hereafter.

[0020] In one possible implementation, the aforementioned vehicle-side interface is also used to display the results of the problem analysis.

[0021] In one possible implementation, the aforementioned system further includes a user terminal for displaying the problem analysis results.

[0022] In one possible implementation, the aforementioned user terminal is further configured to acquire input data, which is obtained based on input operations performed by the user on the user terminal's interactive interface.

[0023] In one possible implementation, the aforementioned vehicle terminal is also used to acquire input data, which is obtained based on the user's input operations on the vehicle terminal's interactive interface.

[0024] In one possible implementation, the aforementioned cloud is specifically used to: take input data as input to a large language model, output question categories and emotional information, the emotional information including the user's emotion corresponding to the input data, the input data including voice data; and output question analysis results and emotional information.

[0025] In one possible implementation, the aforementioned large language model is used to filter out question categories from multiple categories in the pattern library.

[0026] In one possible implementation, the aforementioned pattern library also includes data rules corresponding to each category, wherein the data rules are data types corresponding to each category;

[0027] The aforementioned cloud platform is also used for: determining the first data rule corresponding to the problem category based on the pattern library; and obtaining multimodal data from the vehicle based on the first data rule, wherein the multimodal data includes data of the type indicated by the first data rule.

[0028] In one possible implementation, the aforementioned pattern library also includes a model type corresponding to each category;

[0029] The aforementioned cloud platform is used to: determine the first model type corresponding to the problem category based on the pattern library; and analyze multimodal data based on the model indicated by the first model type to obtain the problem analysis results.

[0030] In one possible implementation, the aforementioned cloud is specifically used for: inputting the problem analysis results into the question-and-answer system, outputting a question answer, wherein the question answer includes the content of the problem analysis results, and the question answer is used to respond to the input data.

[0031] In one possible implementation, the aforementioned cloud platform is also used to: generate a problem description based on the problem analysis results, the problem description including the problem analysis results and the analysis path, the analysis path being used to indicate the cause of the problem indicated by the input data; if the large language model also outputs sentiment information, the problem description also includes sentiment information.

[0032] In one possible implementation, the aforementioned cloud platform is also used to: acquire historical problem information, which includes information related to the problem corresponding to the input data; and analyze the historical problem information based on the problem category and multimodal data to obtain problem information.

[0033] Thirdly, this application provides an operation and maintenance system, including:

[0034] The input module is used to acquire input data, which is obtained based on user input and indicates how to solve vehicle problems.

[0035] The large model analysis module is used to take input data as input to a large language model and output the question category.

[0036] A multimodal data acquisition module is used to acquire multimodal data, which includes data collected by at least one sensor and data related to vehicle problems.

[0037] The multimodal analysis module is used to analyze multimodal data based on problem categories to obtain problem analysis results;

[0038] The output module is used to output the problem analysis results.

[0039] The effects achieved by the third aspect or any optional implementation of the third aspect of this application can be referred to the description of the first aspect or any optional implementation of the first aspect above, and will not be repeated hereafter.

[0040] In one possible implementation, the aforementioned large model analysis module is used to: take the input data as input to the large language model and output the question category and emotion information, wherein the emotion information includes the user emotion corresponding to the input data and the input data includes voice data;

[0041] The aforementioned output module is used to output the problem analysis results and sentiment information.

[0042] In one possible implementation, the aforementioned large language model is used to filter out question categories from multiple categories in the pattern library.

[0043] In one possible implementation, the aforementioned pattern library further includes data rules corresponding to each category, and the data rules are data types corresponding to each category; the aforementioned multimodal data acquisition module is used to: determine the first data rule corresponding to the problem category according to the pattern library; acquire multimodal data according to the first data rule, wherein the multimodal data includes data of the type indicated by the first data rule.

[0044] In one possible implementation, the aforementioned pattern library also includes a model type corresponding to each category; the aforementioned multimodal analysis module is used to: determine the first model type corresponding to the problem category according to the pattern library; and analyze the multimodal data according to the model indicated by the first model type to obtain the problem analysis results.

[0045] In one possible implementation, the aforementioned output module is used to: input the problem analysis results into the question-and-answer system, output a question answer, the question answer including the content of the problem analysis results, and the question answer being used to respond to the input data.

[0046] In one possible implementation, the aforementioned output module is further configured to: generate a problem description based on the problem analysis results, the problem description including the problem analysis results and the analysis path, the analysis path being used to indicate the cause of the problem indicated by the input data; if the large language model also outputs sentiment information, the problem description also includes sentiment information.

[0047] In one possible implementation, the aforementioned system further includes:

[0048] The history module is used to retrieve historical issue information, which includes information related to the issues corresponding to the input data.

[0049] The multimodal analysis module is used to analyze the problem based on the problem category and multimodal data, combined with historical problem information, to obtain problem information.

[0050] Fourthly, embodiments of this application provide a vehicle including a processor and a memory, wherein the processor and the memory are interconnected via a circuit, and the processor calls program code in the memory to perform processing-related functions in the method shown in any of the first aspects above.

[0051] Fifthly, embodiments of this application provide a cloud server, which includes a processor and a memory, wherein the processor and the memory are interconnected via a circuit, and the processor calls program code in the memory to perform processing-related functions in the method shown in any of the first aspects above.

[0052] In a sixth aspect, embodiments of this application provide a user terminal, which includes a processor and a memory, wherein the processor and the memory are interconnected via a circuit, and the processor calls program code in the memory to perform input or output related functions in the method shown in any of the first aspects above.

[0053] In a seventh aspect, embodiments of this application provide a digital processing chip or chip, the chip including a processing unit and a communication interface, the processing unit obtaining program instructions through the communication interface, the program instructions being executed by the processing unit, the processing unit being used to perform processing-related functions as described in the first aspect or any optional embodiment of the first aspect.

[0054] Eighthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the method described in the first aspect or any optional implementation thereof.

[0055] Ninthly, embodiments of this application provide a computer program product comprising a computer program / instructions, which, when executed by a processor, causes the processor to perform the method described in the first aspect or any optional implementation thereof. Attached Figure Description

[0056] Figure 1 is a schematic diagram of a vehicle structure provided in an embodiment of this application;

[0057] Figure 2 is a schematic diagram of the structure of an operation and maintenance system provided in an embodiment of this application;

[0058] Figure 3 is a flowchart illustrating a data analysis method provided in an embodiment of this application;

[0059] Figure 4 is a schematic diagram of another operation and maintenance system provided in an embodiment of this application;

[0060] Figure 5 is a schematic diagram of another operation and maintenance system provided in an embodiment of this application;

[0061] Figure 6 is a schematic diagram of another operation and maintenance system provided in an embodiment of this application;

[0062] Figure 7 is a schematic diagram of another operation and maintenance system provided in an embodiment of this application;

[0063] Figure 8 is a schematic diagram of another operation and maintenance system provided in an embodiment of this application;

[0064] Figure 9 is a flowchart illustrating another data analysis method provided in an embodiment of this application;

[0065] Figure 10 is a schematic diagram of a GUI provided in an embodiment of this application;

[0066] Figure 11 is another GUI schematic diagram provided in an embodiment of this application;

[0067] Figure 12 is another GUI schematic diagram provided in an embodiment of this application;

[0068] Figure 13 is another GUI schematic diagram provided in an embodiment of this application;

[0069] Figure 14 is a schematic diagram of an application scenario provided by an embodiment of this application;

[0070] Figure 15 is a schematic diagram of an application scenario provided by an embodiment of this application;

[0071] Figure 16 is a flowchart illustrating another data analysis method provided in an embodiment of this application;

[0072] Figure 17 is a flowchart illustrating another data analysis method provided in an embodiment of this application;

[0073] Figure 18 is another GUI schematic diagram provided in an embodiment of this application;

[0074] Figure 19 is another GUI schematic diagram provided in an embodiment of this application;

[0075] Figure 20 is another GUI schematic diagram provided in an embodiment of this application;

[0076] Figure 21 is another GUI schematic diagram provided in an embodiment of this application;

[0077] Figure 22 is a schematic diagram of another operation and maintenance system provided in an embodiment of this application;

[0078] Figure 23 is a schematic diagram of another operation and maintenance system provided in an embodiment of this application;

[0079] Figure 24 is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0080] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0081] The method provided in this application can be applied to various smart terminals, such as vehicles, robots, or other electronic devices.

[0082] The following describes the device structure of the method provided in this application embodiment, taking deployment in a vehicle as an example. For instance, it can be applied to obstacle avoidance scenarios when there are emergency traffic situations ahead of the vehicle, such as when a vehicle overturns or loses control, to plan a safer driving path for the vehicle; or it can be deployed in a robot to plan a more reasonable obstacle avoidance path for the robot when there are moving obstacles in front of it. This application exemplifies the deployment of the method provided in this application embodiment in a vehicle as an example.

[0083] Referring to Figure 1, which is a schematic diagram of a vehicle structure provided in an embodiment of this application, the vehicle 100 can be configured in an intelligent driving mode. For example, the vehicle 100 can control itself while in intelligent driving mode, and can determine the current state of the vehicle and its surrounding environment through human operation, determine whether there are obstacles in the surrounding environment, and control the vehicle 100 based on the obstacle information. When the vehicle 100 is in intelligent driving mode, it can also be set to operate without human interaction.

[0084] Figure 1 is a functional block diagram of a vehicle 100 provided in an embodiment of this application. The vehicle 100 can be configured to a full or partial intelligent driving mode. For example, the vehicle 100 can obtain environmental information about its surroundings through the perception system 120, and obtain an intelligent driving strategy based on the analysis of the surrounding environmental information to achieve full intelligent driving, or present the analysis results to the user to achieve partial intelligent driving.

[0085] Vehicle 100 may include various subsystems, such as an infotainment system 110, a perception system 120, a decision control system 130, a drive system 140, and a computing platform 150. Optionally, vehicle 100 may include more or fewer subsystems, and each subsystem may include multiple components. In addition, each subsystem and component of vehicle 100 may be interconnected via wired or wireless means.

[0086] In some embodiments, the infotainment system 110 may include a communication system 111, an entertainment system 112, and a navigation system 113.

[0087] Communication system 111 may include wireless communication system 111, which can communicate wirelessly with one or more devices directly or via a communication network. For example, wireless communication system 111 may use 3G cellular communication, such as CDMA, EVDO, GSM / GPRS, or 4G cellular communication, such as LTE, or 5G cellular communication. Wireless communication system 111 may communicate using WiFi and wireless local area network (WLAN). In some embodiments, wireless communication system 111 may communicate directly with devices using infrared links, Bluetooth, or ZigBee. Wireless communication system 111 may include one or more dedicated short range communications (DSRC) devices, which may include public and / or private data communications between vehicles and / or roadside stations.

[0088] In this embodiment of the application, communication between the vehicle and the cloud can be achieved through the communication system 111.

[0089] The entertainment system 112 may include a central control screen, a microphone, and speakers. Users can listen to the radio and play music within the vehicle using the entertainment system 112; or connect their mobile phones to the vehicle and project their screens onto the central control screen, which may be touch-sensitive, allowing users to operate the system. In some cases, the microphone can capture the user's voice signal, and analysis of this signal can enable the user to control certain aspects of the vehicle 100, such as adjusting the interior temperature. In other cases, music can be played to the user through the speakers.

[0090] The interactive interface provided in this application embodiment can be displayed in the infotainment system. Specifically, it can be displayed on devices in the vehicle, such as the central control screen, passenger screen, instrument panel screen, and head-up display, which can be used to display interactive interfaces to users. Users can perform interactive operations based on the interactive interface, which may include, but are not limited to, touch interaction, voice interaction, and gesture interaction, depending on the actual interaction methods supported by the vehicle.

[0091] The interactive interface mentioned below in this application may also include the interactive interface in a user terminal, where users can perform interactive operations and feedback content can be displayed to users through the interactive interface of the terminal.

[0092] Alternatively, in some scenarios, the vehicle can connect to the user terminal, as shown in Figure 2 below. For example, the user can interact with the vehicle's interface, and feedback content can be displayed on the user terminal's screen. Or, the user can interact with the terminal's interface, and feedback content can be displayed on the vehicle's interface, thus achieving linkage between the vehicle and the user terminal and improving the user experience.

[0093] The navigation system 113 may include map services to provide navigation for the vehicle 100, and the navigation system 113 may be used in conjunction with the vehicle's global positioning system 121 and inertial measurement unit 122. The map may be a two-dimensional map, a high-precision map, or a map constructed based on data collected during the vehicle's operation.

[0094] The perception system 120 may include several sensors for sensing information about the environment surrounding the vehicle 100. For example, the perception system 120 may include a global positioning system 121 (which may include a global navigation satellite system (GNSS), specifically GPS, BeiDou, or other positioning systems), an inertial measurement unit (IMU) 122, a lidar (LiDAR) 123, a millimeter-wave radar 124, an ultrasonic radar 125, and a camera device 126. The perception system 120 may also include sensors from the internal systems of the monitored vehicle 100 (e.g., an in-vehicle air quality monitor, fuel gauge, oil temperature gauge, etc.). Sensor data from one or more of these sensors can be used to detect objects and their corresponding characteristics (position, shape, orientation, speed, etc.). This detection and identification is a key function for the safe operation of the vehicle 100. The multimodal data from the vehicle mentioned below may include data collected by various units in the perception system 120, such as point cloud data collected by lidar or environmental images collected by the camera device.

[0095] The Global Positioning System 121 can be used to determine the geographical location of vehicle 100.

[0096] The inertial measurement unit 122 is used to sense changes in the position and orientation of the vehicle 100 based on inertial acceleration. In some embodiments, the inertial measurement unit 122 may be a combination of an accelerometer and a gyroscope.

[0097] The lidar 123 can use lasers to sense objects in the environment in which the vehicle 100 is located. In some embodiments, the lidar 123 may include one or more laser sources, a laser scanner, and one or more detectors, as well as other system components.

[0098] The millimeter-wave radar 124 can use radio signals to sense objects in the surrounding environment of the vehicle 100. In some embodiments, in addition to sensing objects, the millimeter-wave radar 124 can also be used to sense the speed and / or direction of travel of objects.

[0099] The ultrasonic radar 125 can use ultrasonic signals to sense objects around the vehicle 100.

[0100] The camera device 126 can be used to capture image information of the surrounding environment of the vehicle 100. The camera device 126 may include a monocular camera, a binocular camera, a structured light camera, and a panoramic camera, etc. The image information acquired by the camera device 126 may include still image information or video stream information.

[0101] The decision control system 130 includes a computing system 131 that analyzes and makes decisions based on information acquired by the sensing system 120. The decision control system 130 also includes a vehicle controller 132 that controls the power system of the vehicle 100, and a steering system 133, a throttle 134, and a braking system 135 for controlling the vehicle 100.

[0102] The computing system 131 can process and analyze various information acquired by the perception system 120 to identify targets, objects, and / or features in the environment surrounding the vehicle 100. The targets may include pedestrians or animals, and the objects and / or features may include traffic signals, road boundaries, and obstacles. The computing system 131 may use object recognition algorithms, Structure from Motion (SFM) algorithms, video tracking, and other techniques. In some embodiments, the computing system 131 may be used to map the environment, track objects, estimate object speeds, etc. The computing system 131 can analyze the acquired information and derive a control strategy for the vehicle.

[0103] The vehicle controller 132 can be used to coordinate the control of the vehicle's power battery and engine 141 to improve the power performance of the vehicle 100.

[0104] The steering system 133 can be used to adjust the forward direction of the vehicle 100. For example, in one embodiment, it can be a steering wheel system.

[0105] The throttle 134 is used to control the operating speed of the engine 141 and thus the speed of the vehicle 100.

[0106] Braking system 135 is used to control the deceleration of vehicle 100. Braking system 135 can use friction to slow down the rotational speed of wheel 144. In some embodiments, braking system 135 can convert the kinetic energy of wheel 144 into electric current. Braking system 135 may also take other forms to slow down the rotational speed of wheel 144 to control the speed of vehicle 100.

[0107] The drive system 140 includes components that provide powered motion to the vehicle 100. In one embodiment, the drive system 140 may include an engine 141, an energy source 142, a transmission system 143, and wheels 144. The engine 141 may be an internal combustion engine, an electric motor, an air-compressed engine, or other types of engine combinations, such as a hybrid engine consisting of a gasoline engine and an electric motor, or a hybrid engine consisting of an internal combustion engine and an air-compressed engine. The engine 141 converts the energy source 142 into mechanical energy.

[0108] Examples of energy sources 142 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity. Energy source 142 may also provide energy to other systems of vehicle 100.

[0109] The drivetrain 143 transmits mechanical power from the engine 141 to the wheels 144. The drivetrain 143 may include a gearbox, a differential, and a drive shaft. In one embodiment, the drivetrain 143 may also include other components, such as a clutch. The drive shaft may include one or more axles that can be coupled to one or more wheels 144.

[0110] Some or all of the functions of vehicle 100 are controlled by computing platform 150. Computing platform 150 may include at least one processor 151, which can execute instructions 153 stored in a non-transitory computer-readable medium such as memory 152. In some embodiments, computing platform 150 may also be multiple computing devices that control individual components or subsystems of vehicle 100 in a distributed manner.

[0111] Processor 151 can be any conventional processor, such as a commercially available CPU. Alternatively, processor 151 may also include a graphics processing unit (GPU), a field-programmable gate array (FPGA), a system-on-chip (SoC), an application-specific integrated circuit (ASIC), or a combination thereof. Processor 151 can be located on a device remote from the vehicle and can communicate wirelessly with the vehicle.

[0112] In some embodiments, memory 152 may contain instructions 153 (e.g., program logic) that can be executed by processor 151 to perform various functions of vehicle 100. Memory 152 may also contain additional instructions, including instructions for sending data to, receiving data from, interacting with, and / or controlling one or more of the infotainment system 110, perception system 120, decision control system 130, and drive system 140.

[0113] In the method provided in this application embodiment, the steps performed by the vehicle can be executed by the computing platform 150. For example, the processor 151 can read the program stored in the memory 152, display the interactive interface on the display screen, and execute subsequent processes based on the user's interactive operation.

[0114] In addition to instruction 153, memory 152 may also store data such as road maps, route information, vehicle position, direction, speed, and other similar vehicle data, as well as other information. It may also store higher-precision maps obtained by the methods provided in this application embodiment. This information can be used by vehicle 100 and computing platform 150 during operation of vehicle 100 in autonomous, semi-autonomous, and / or manual modes.

[0115] The computing platform 150 can control the functions of the vehicle 100 based on inputs received from various subsystems, such as the drive system 140, the perception system 120, and the decision control system 130. For example, the computing platform 150 can utilize inputs from the decision control system 130 to control the steering system 133 to avoid obstacles detected by the perception system 120. In some embodiments, the computing platform 150 is operable to provide control over many aspects of the vehicle 100 and its subsystems.

[0116] Alternatively, one or more of these components may be installed separately from or associated with vehicle 100. For example, memory 152 may exist partially or completely separately from vehicle 100. The components may be communicatively coupled together in a wired and / or wireless manner.

[0117] Optionally, the above components are just an example. In actual applications, the components in the above modules may be added or deleted according to actual needs. Figure 1 should not be construed as a limitation on the embodiments of this application.

[0118] The aforementioned vehicle 100 can be any vehicle or vehicle-mounted terminal capable of intelligent driving, such as a car, truck, motorcycle, bus, ship, airplane, helicopter, recreational vehicle, amusement park vehicle, construction equipment, tram, golf cart, or train. This application embodiment does not impose any special limitations on this type of vehicle.

[0119] Based on the aforementioned vehicle architecture, the method flow provided in the embodiments of this application will be described below.

[0120] As the number of autonomous vehicles increases, problems during their use are gradually emerging. Issues arising during the use of autonomous vehicles can typically be addressed by user feedback based on the specific circumstances, with maintenance personnel manually locating the vehicle malfunction or the user-reported problem. However, this manual location approach is costly, leading to the development of more intelligent maintenance systems.

[0121] In intelligent operation and maintenance systems, voice feedback is one of the important means of interaction between users and the system. However, the content of voice feedback is often ambiguous, the amount of feedback is huge, and there is a lack of effective prioritization. Therefore, how to automatically process and analyze voice feedback has become a major challenge.

[0122] For example, in one existing solution, fault information is matched with voice information through audio profiles, and the voice information is played according to priority flags, which can help maintenance personnel quickly understand vehicle fault information. This is equivalent to pre-setting some audio configuration information; when a user reports a problem or detects a vehicle fault via voice, the appropriate voice feedback can be selected from the audio configuration to provide the user. However, its fault information recognition mainly relies on predefined audio profiles, and its understanding and processing of user subjective feedback is relatively limited, lacking in-depth analysis of subtle differences in user language. Prioritization handling in multi-fault situations is relatively simple and difficult to handle priority conflicts in complex fault environments. Furthermore, it cannot effectively handle ambiguous user voice descriptions, leading to omissions in the identification of potential problems. In addition, this solution focuses on voice broadcasting and fails to conduct in-depth comprehensive diagnosis, which may result in misdiagnosis.

[0123] For example, one existing solution uses deep learning technology to automatically analyze user-uploaded fault images to determine the specific fault type of the vehicle and provide maintenance personnel with repair suggestions. However, this solution has limited utilization of voice feedback, focusing only on the analysis of image faults and lacking comprehensive analysis and understanding of the semantic information described by the user. Fault identification mainly relies on image features, and the system's fault type discrimination is relatively simple in multi-fault scenarios, limiting the types of faults that can be identified.

[0124] Clearly, traditional voice-assisted maintenance solutions typically rely on keyword matching and simple semantic understanding, making it difficult to accurately identify and classify user-described problems. This is especially true in scenarios involving complex issues, where these solutions struggle to provide effective automation. Furthermore, traditional solutions have limited utilization of vehicle multimodal data, resulting in insufficient accuracy and coverage of diagnostic results.

[0125] Therefore, this application provides a data analysis method that, by introducing a large language model and multimodal data analysis technology, overcomes the shortcomings of traditional methods in speech understanding, data fusion, and problem diagnosis, and can more accurately classify problems, thereby improving the accuracy and coverage of problem diagnosis.

[0126] First, the method provided in this application embodiment can be directly deployed on the vehicle, or it can be deployed on both the vehicle and the cloud in a collaborative manner. Optionally, when deploying the method provided in this application embodiment on the cloud and the vehicle, some processes can be further deployed on user terminals.

[0127] In one possible implementation, the method steps mentioned in the embodiments of this application can be executed by a vehicle. For example, user input data can be collected at the vehicle's interactive interface, and the collected data from various sensors can be used locally on the vehicle to perform problem analysis, with the analysis results displayed to the user in the interactive interface.

[0128] In one possible implementation, the method steps mentioned in the embodiments of this application can be executed collaboratively by the vehicle and the cloud, that is, some method steps are deployed on the vehicle and others are deployed on the cloud. For example, as shown in Figure 2, this application also provides an operation and maintenance system 20, which can deploy a user interface on the vehicle 23. Users can perform interactive operations through the interface, which may include data input operations. The vehicle can collect the input data input by the user through the user interface. The vehicle 23 can send the user's input data to the cloud 22, and can also send data collected by various sensors on the vehicle 23. After analyzing the received data, the cloud provides feedback on the problem analysis results to the vehicle, and the vehicle displays the problem analysis results to the user through the interactive interface.

[0129] Optionally, some steps can be deployed on the user terminal 24. For example, the user can interact with the interface of the user terminal 24, which acquires the user's input data and sends it to the cloud. Simultaneously, the vehicle terminal 23 can proactively or upon request from the cloud send data collected by its sensors to the cloud. The cloud can provide feedback on the problem analysis results to the user terminal, which then displays these results to the user through its interface. Alternatively, the user can interact with the interface of the vehicle terminal 23, while the problem analysis results are displayed on the interface of the user terminal 24. Or, the user can interact with the interface of the user terminal 24, while the problem analysis results are displayed on the interface of the vehicle terminal 24.

[0130] The method flow provided in the embodiments of this application will be described below first.

[0131] Referring to Figure 3, a flowchart of a data analysis method provided in an embodiment of this application is shown below.

[0132] 301. Obtain input data.

[0133] The input data can be obtained based on user input and can be used to instruct on how to resolve vehicle problems, which are problems encountered by the user while using the device.

[0134] The input data can be voice data, text, or gestures entered by the user, and the specific type can be determined based on the user's input method.

[0135] The device may specifically include the aforementioned smart terminal, such as a vehicle, robot, or other electronic device. This application, as an example, describes the device as a smart car.

[0136] In some scenarios, when a vehicle malfunctions, its intelligent driving functions become abnormal, or the user needs to provide suggestions, the user can use voice input through the interactive interface to indicate the problem. For example, if an abnormality occurs while using the vehicle's intelligent driving function, such as abnormal lane changing or abnormal braking, the user can report the problem to the maintenance system through the interactive interface, requesting the maintenance system to resolve or address the issue. As another example, if a user intends to provide suggestions regarding the vehicle's intelligent driving function, they can provide these suggestions through the interactive interface, requesting the maintenance system to update the feedback. In this embodiment, it can be assumed that the user's input data includes the issue to be reported. When the user needs to provide suggestions, the intended suggestion can be considered as the issue; further details will not be elaborated upon below.

[0137] For example, in a car equipped with intelligent driving functions, the owner can trigger problem feedback through the system's voice prompts. The owner describes the problem, and the system collects voice data and other vehicle information. With the owner's consent, this information is uploaded to the cloud after privacy protection measures are taken. During this process, the system not only collects voice information but can also, with the owner's permission, simultaneously collect the vehicle's current status, location information, and environmental data to provide background support for subsequent analysis.

[0138] 302. Use the input data as input to the large language model and output the question category.

[0139] Specifically, when the input data includes text, it can be directly used as input to the large language model. When the input data is of other types, such as speech data, it can be converted into text and then used as input to the large language model, outputting the problem category. Essentially, the large language model is used to identify the problem to be solved, thus determining its category.

[0140] Optionally, the large language model can also output emotion information, which can be used to indicate the user's emotion corresponding to the input data, specifically represented by a score or emotion category. For example, when the input data is voice data, the user may be conveying emotions such as excitement or anger when inputting the voice data. The large language model can output the user's emotion information to prioritize questions based on the user's emotions, thereby providing prompt feedback and answers to the user's questions, calming the user's emotions in a timely manner, and improving the user experience.

[0141] Optionally, a pattern library can be deployed, which may include multiple problem categories and specific information under each problem category. The specific information under each problem category may include, but is not limited to:

[0142] The data rules for each category indicate the data collection rules when analyzing problems of each category, such as one or more types of data.

[0143] Alternatively, the classification criteria or keywords for each question category;

[0144] Alternatively, there could be available model categories for each question category, or response templates for each question.

[0145] Correspondingly, if a pattern library is deployed, the large model can determine the problem category corresponding to the input data based on the pattern library. For example, it can select the appropriate problem category from the various problem categories included in the pattern library and output it.

[0146] Therefore, in this embodiment of the application, various problem categories can be set more flexibly through the pattern library, so that when adding or adjusting problems, the pattern library can be directly adjusted, thereby adapting to more scenarios and having stronger generalization.

[0147] Step 302 can be executed by the vehicle or by the cloud, depending on the specific application scenario.

[0148] 303. Obtain multimodal data.

[0149] The multimodal data can include data collected by one or more types of sensors. For example, during vehicle operation, sensors installed in the vehicle can collect environmental information within a certain range in real time.

[0150] For example, the multimodal data may include, but is not limited to, data collected by sensors such as image sensors (or cameras), infrared sensors, lidar, or millimeter-wave radar. Accordingly, the multimodal data may include images, point cloud data, or infrared data.

[0151] Furthermore, the multimodal data can specifically be data related to a problem category. For example, the multimodal data can specifically include one or more types of data generated during the period in which the problem is to be solved, or the multimodal data can include data of a type related to the problem to be solved.

[0152] For example, if the problem to be solved is an abnormal lane change problem, the multimodal data can include data collected by multiple sensors in the vehicle within a certain time range before and after the vehicle makes an abnormal lane change.

[0153] For example, if the problem to be solved is a parking anomaly, the multimodal data may include images or point cloud data collected during the parking process to represent changes in the vehicle's environment, or other types of data related to vehicle parking.

[0154] Optionally, if the method provided in this application embodiment further deploys a pattern library, after determining the problem category, it can also be based on the data rules corresponding to each problem category in the pattern library. These data rules can indicate data collection rules related to the problem category. When acquiring multimodal data, the corresponding data rules can be determined according to the problem category. For ease of distinction, these rules can be referred to as the first data rules. Multimodal data related to the problem category is acquired according to these first data rules, thereby obtaining data related to the problem to be solved.

[0155] Furthermore, if step 303 is deployed on the vehicle side, the vehicle side can directly read multimodal data from the storage space, or perform multimodal data collection, etc. If step 303 is deployed in the cloud, the cloud can receive multimodal data uploaded by the vehicle side.

[0156] For example, in one possible implementation, if step 303 is deployed in the cloud, after obtaining the problem category in the cloud, the required multimodal data can be obtained based on that problem category. Specifically, this could be obtaining multimodal data according to the aforementioned data rules, or obtaining multimodal data that is temporally related to the problem to be solved, etc., depending on the actual application scenario.

[0157] 304. Analyze the problem based on the problem category and multimodal data to obtain the problem analysis results.

[0158] Once the problem category and corresponding multimodal data are determined, the multimodal data and problem category can be used to conduct a specific analysis of the problem to obtain the problem analysis results. These results can include information directly related to the problem or the causes of the problem. For example, analyzing the causes, frequency, location, or latitude and longitude of the problem.

[0159] Optionally, the problem analysis result can also be determined by combining historical problem analysis results or historical problem feedback information. For example, historical problems related to the current problem can be queried based on the problem category or the location corresponding to the problem, and the historical problem analysis results or historical problem feedback information corresponding to the historical problem can be obtained. Subsequently, when conducting problem analysis, the content of the historical problem can be combined with the problem analysis, or the problem analysis result can be updated using the content of the historical problem after obtaining the initial problem analysis result. For example, when analyzing the current problem, the problem analysis result of the current problem can be generated by combining the historical problem analysis results. Alternatively, after obtaining the initial problem analysis result based on the problem category and multimodal data, the problem analysis result can be updated using the historical problem analysis results or historical problem feedback information, and the updated problem analysis result can be output.

[0160] Optionally, the model type corresponding to the problem category can be determined based on the pattern library. For ease of distinction, this can be referred to as the first model type. For example, different models are required for processing data of different modalities. Based on the first model type corresponding to the problem category, a model type suitable for the acquired multimodal data can be selected. Subsequently, the multimodal data is analyzed according to the model indicated by the first model type to obtain the problem analysis results. Therefore, in this embodiment, for data of different modalities, a suitable model can be selected for processing, thereby outputting more accurate problem analysis results.

[0161] 305. Output the problem analysis results.

[0162] Once the problem analysis results are obtained, output can be generated based on those results.

[0163] If step 304 is deployed in the cloud, after obtaining the problem analysis results, the results can be sent to the vehicle terminal to output the results to the user; alternatively, the results can be output to maintenance personnel in the cloud.

[0164] In this embodiment, a large language model can be used to analyze user input data more flexibly and accurately, and to accurately identify the problem category to be solved. Furthermore, multimodal data collected from the vehicle end can be used to perform more precise analysis of the problem, outputting specific information related to the problem, such as the cause or frequency of the problem, to provide users with more accurate operational and maintenance requirements.

[0165] Typically, feedback can be provided to users (or car owners) and maintenance personnel based on the results of the problem analysis. The content of the feedback may differ depending on the person being addressed.

[0166] In one possible implementation, when providing feedback to a user, the problem analysis results can be input into a question-and-answer system, and a question answer can be output. The question answer includes the content of the problem analysis results and is used to respond to the input data. The question answer is output through an interactive interface, which includes an interface determined based on the input data, thereby providing users with more intuitive feedback in a question-and-answer format.

[0167] In one possible implementation, when providing feedback to operations and maintenance personnel, a problem description can be generated based on the problem analysis results. The problem description includes the problem analysis results and analysis path, with the analysis path indicating the cause of the problem indicated by the input data. If the large language model also outputs sentiment information, the problem description also includes sentiment information, so that operations and maintenance personnel can determine the priority of the user's requested problem based on the user's sentiment information, and can promptly appease the user and improve the user experience.

[0168] The foregoing has described the method flow provided in the embodiments of this application. The following is a detailed introduction in conjunction with specific application scenarios, the operation and maintenance system provided in the embodiments of this application, and the specific interactive interface.

[0169] Referring to Figure 4, this application provides an architecture diagram of an operation and maintenance system.

[0170] The operation and maintenance system may include a voice acquisition module 401, a voice analysis module 402, a multimodal data acquisition module 403, a multimodal analysis module 404, a problem fusion module 405, a feedback generation module 406, and an interaction module 407, etc.

[0171] The voice acquisition module 401 can be used to acquire voice data input by the user. Of course, the user can also use other types of input data in the system provided in this application embodiment, and the voice acquisition module can also be replaced with other types of input modules. Here, the example of input voice data is used for illustration. The semantic acquisition module can be replaced according to the actual application scenario. This application will not elaborate on this.

[0172] The speech analysis module 402 can be used to analyze the collected speech data, output the corresponding question category, and further output user emotion information, etc. The analysis method may differ for different types of input data. This speech analysis module can also be replaced by other analysis modules, such as a large model analysis module.

[0173] The multimodal data acquisition module 403 can be used to acquire data collected by one or more sensors in the vehicle.

[0174] The multimodal analysis module 404 can be used to further analyze the problem to be solved based on multimodal data and output the problem analysis results.

[0175] The problem fusion module 405 can be used to fuse the speech analysis results with the analysis results of multimodal data and output the final problem analysis results.

[0176] The feedback generation module 406 can be used to output user-friendly problem description reports based on the problem analysis results.

[0177] The interaction module 407 can be used to provide users with an interactive interface and to provide users with feedback such as problem descriptions and reports through the interactive interface.

[0178] The history module 408 can be used to acquire data on historical issues, including problem analysis results or feedback information. The history module 408 is optional; when deployed, the multimodal problem module 404 can also combine the historical problem data acquired by the history module 408, such as problem analysis results or feedback information, to output problem analysis results.

[0179] The aforementioned modules can be deployed adaptively according to the actual scenario.

[0180] For example, as shown in Figure 5, the voice acquisition module 401, voice analysis module 402, multimodal data acquisition module 403, multimodal analysis module 404, question fusion module 405, feedback generation module 406, interaction module 407, and history module 408 can all be directly deployed in the vehicle, that is, the method provided in the embodiments of this application is executed in the vehicle.

[0181] For example, as shown in Figure 6, the voice acquisition module 401 and the interaction module 407 can be deployed on the vehicle side, while the voice analysis module 402, the multimodal data acquisition module 403, the multimodal analysis module 404, the question fusion module 405, and the feedback generation module 406 can be deployed in the cloud. That is, data acquisition and interaction with the user are performed on the vehicle side, while data processing steps are performed in the cloud.

[0182] For example, as shown in Figure 7, the voice acquisition module 401, voice analysis module 402, and interaction module 407 can be deployed on the vehicle side, while the multimodal data acquisition module 403, multimodal analysis module 404, question fusion module 405, and feedback generation module 406 can be deployed in the cloud. That is, data acquisition, voice analysis, and user interaction are performed on the vehicle side, while steps related to multimodal data processing are performed in the cloud.

[0183] Furthermore, in the aforementioned system architecture that includes a user terminal, the interaction module can be deployed in the user terminal, or the interaction module 407 can be further divided into multiple sub-modules, as shown in Figure 8, which can be divided into interaction sub-module 4071 and interaction sub-module 4072. Interaction sub-module 4071 can be deployed on the vehicle side, and interaction sub-module 4072 can be deployed in the user terminal.

[0184] Correspondingly, a voice acquisition module 409 can also be set in the user terminal. Its function is similar to that of the aforementioned voice acquisition module 401, and will not be described in detail here.

[0185] The following section will provide a detailed introduction to the functions of each module in the operation and maintenance system, based on the aforementioned methodology.

[0186] Referring to Figure 9, a flowchart of another data analysis method provided in this application embodiment is shown below.

[0187] 901. Voice reporting issue.

[0188] After the vehicle is started, the display screen inside the vehicle can show the user an interactive interface where the user can perform various operations. For example, the user can select voice interaction in the interactive interface to input voice data and report problems to be solved via voice.

[0189] For example, as shown in Figure 10, a vehicle-side GUI provided in this application embodiment can display a "Report Problem" button for the user. The user can report problems that occur during vehicle driving by interacting with this button. After the user selects "Report Problem," as shown in Figure 11, various input methods can be displayed for the user, such as voice or text input. This application takes voice input as an example. When the user makes a voice input, the voice data input by the user can be collected. For example, the user can input "Why do I always brake suddenly?", "Why do I always change lanes suddenly?", "Why do I always change lanes when there is a car in the next lane?", "The speed is too fast," etc.

[0190] For example, as shown in Figure 12, a user terminal GUI provided in this application embodiment can display a "Report Problem" button for the user. The user can report problems occurring during vehicle driving by interacting with this button. After the user selects "Report Problem," further as shown in Figure 13, various input methods can be displayed for the user, such as voice or text input.

[0191] Of course, the interactive interface shown in this application only introduces the meaning of some of the buttons. The actual button deployment method can be determined according to the actual application scenario, and this application does not limit it.

[0192] This can be understood as follows: when a car equipped with an intelligent driving system is in use, the owner triggers feedback through the system's voice prompts. The owner describes the problem, and the system collects voice data and other vehicle information. This information, after obtaining the owner's consent, is uploaded to the cloud after privacy protection. During this process, the system not only collects voice information but also simultaneously collects the vehicle's current status, location information, and environmental data to provide background support for subsequent analysis.

[0193] Furthermore, when sending data from the vehicle to the cloud, data can be sent with the user's permission. For example, a dialog box can be displayed in the interface asking the user whether they allow data upload. After the user checks their consent, the vehicle can upload the data to the cloud. Alternatively, the user can pre-enable the permission to report data to the cloud in the device's authorization list, thereby enhancing the security of vehicle-side data.

[0194] 902. Large language models are used for problem classification.

[0195] In other words, the aforementioned speech analysis module 402 can use a large language model for analysis and output specific information about the problem to be solved. For example, it can include the problem classification, possible causes of the problem, or analysis paths.

[0196] Optionally, the large language model can also output a confidence score, which can be used to represent the accuracy of the output question category.

[0197] The input to a large language model can be speech data or text obtained by converting speech data.

[0198] Optionally, a pattern library can be pre-deployed. The data in this library can be categorized into: a category list, data rules, model categories, and response templates. The category list can include one or more question categories, such as parking, collision, and intelligent driving issues. This list is generated by human experts or through algorithmic clustering based on historical customer feedback. Data rules can include the data collection or retrieval rules for each question category, or the required data types. Model categories include the types of models used in each question category. Response templates can be templates for responding to users in each question category.

[0199] In the process of outputting question categories by a large language model, the output can be based on the question categories included in the pattern library. For example, if the user's input voice is "Why does the car always brake suddenly?", the large language model can understand the semantics of the input voice data and select the question category that is more suitable for the voice data based on the question categories in the pattern library. For example, the problem to be solved can be classified as an intelligent driving function problem.

[0200] Step 902 can be deployed on the vehicle or in the cloud. When step 902 is deployed in the cloud, the vehicle can report voice data to the cloud, and the cloud uses a large language model to classify the voice data into questions. Correspondingly, the pattern library can also be deployed in the cloud.

[0201] Furthermore, large language models can also identify user emotional information based on input speech data. This emotional information can be used to provide feedback on the user's emotions when reporting a problem. Specifically, this emotional information can include the user's emotional category or a score of their emotions. For example, different tones or wording may express different emotions when a user reports a problem via voice. For instance, a user might use tones or wording expressing tension, anger, surprise, or indifference. A large language model can be used to identify the user's emotions based on the speech data and output the corresponding emotional information. This emotional information can directly include the user's emotional category or a score of their emotions. This emotional information can be used to determine the priority of the problem to be solved. For example, if the user is angry or surprised, it indicates that the problem urgently needs to be solved, and the corresponding priority is higher. Alternatively, if the emotional information includes an emotional score, a higher score indicates a higher urgency of solving the problem. Therefore, in this embodiment, a large model can be used to identify user emotions, thereby determining the priority of the problem to be solved based on the user's emotions. This allows for timely feedback of problem analysis results to the user, enabling them to obtain relevant information about the problem promptly and improving the user experience. In other words, through voice analysis, it is possible not only to identify the specific questions raised by users, but also to judge the emotional state of users, thereby further improving the quality of interaction between the system and users.

[0202] For example, large language models can typically be deployed in the cloud. The deployment process for a large language model can be shown in Figure 14. The initial structure of the large language model can be obtained, and speech questions and pattern libraries can be collected as training sets. The large language model can be fine-tuned using speech questions and pattern libraries. The fine-tuned large language model can then be deployed in the cloud. When the user's speech data is input into the large language model, the question category can be output.

[0203] Therefore, in this embodiment of the application, the large language model can be used to delve deeper into the hidden information in the user's description, thereby better understanding the user's real needs and the specific problems encountered.

[0204] 903. Determine whether the confidence level of the problem classification result is higher than the first threshold. If yes, use the output result of the large model as the problem analysis result and proceed to step 905. If no, proceed to step 904.

[0205] Step 903 is an optional step, meaning that step 903 can be skipped and step 904 can be executed directly.

[0206] This can be understood as follows: when the output of the large language model can solve the problem or provide an answer to the problem, feedback can be directly provided to the user based on the output of the large language model, thereby improving the efficiency of providing feedback to the user.

[0207] 904. Obtain multimodal data.

[0208] This multimodal data can include images captured by the vehicle's cameras, LiDAR data, vehicle speed, acceleration, and other information.

[0209] The collection rules for this multimodal data can be based on pre-defined rules or on data rules corresponding to the problem classification in the pattern library.

[0210] For example, after determining the problem category, multimodal data can be obtained during the time period in which the problem occurred, according to pre-set rules. This includes data collected by various sensors in the vehicle.

[0211] For example, data rules corresponding to problem categories can be read from the pattern library. These data rules can include data types corresponding to problem categories, and data collected by one or more sensors in the vehicle can be obtained based on these data rules.

[0212] If step 904 is executed by the vehicle, the vehicle can directly read multimodal data from the stored data; if step 903 is executed by the cloud, the cloud may request multimodal data from the vehicle, and the vehicle may report the multimodal data required by the cloud to the cloud, or the vehicle may simultaneously report the multimodal data collected by the vehicle when reporting voice data.

[0213] Typically, the type of data to be collected may differ depending on the type of problem. For example, in a scenario involving vehicle control anomalies, it may be necessary to collect vehicle control data and environmental data; for a specific functional anomaly in a vehicle, it may only be necessary to collect the vehicle's internal function logs. The specific type of multimodal data required can be determined based on the actual application scenario.

[0214] 905. Multimodal data analysis.

[0215] Once the multimodal data is obtained, it can be analyzed to further determine information relevant to the problem to be solved.

[0216] Optionally, a multimodal large model can be used for multimodal data analysis. This model can combine data from multiple modalities for analysis, outputting problem analysis results, including a description of the problem, the causes of the problem, or the analysis path leading to the problem. Specifically, multimodal data can be input into the multimodal large model, with different modalities input to different input ports. This allows the multimodal large model to process data from different modalities separately, and internally it can fuse multimodal data features, thereby combining multiple types of data features to more accurately analyze the problem.

[0217] For example, the final problem categories analyzed can be as shown in Figure 15. For instance, the problem categories in the pattern library can be divided into Category 1: Parking; Category 2: Collision; ... Category x: ... and so on.

[0218] Therefore, in this embodiment, the multimodal data includes images captured by the vehicle's camera, LiDAR data, vehicle speed, acceleration, etc. Through multimodal data, the operation and maintenance system can further verify and supplement the information in the voice feedback. For example, the camera images can be used to verify the environmental conditions described by the user, the LiDAR data can be used to confirm the distance between the vehicle and obstacles, and the speed and acceleration information can be used to judge the dynamic performance of the vehicle when the problem occurs, thereby comprehensively improving the accuracy of the analysis.

[0219] Furthermore, if the large language model outputs specific information about the problem to be solved in step 902 above, the specific information about the problem to be solved output by the large language model can be fused with the specific information output based on multimodal data to output the final problem analysis result.

[0220] For example, the analysis process of the problem to be solved can be shown in Figure 16. Taking the collision problem as an example, a model corresponding to the collision category is selected from the pattern library. If a collision model is selected, the multimodal data can be checked and judged. The large model is used to analyze the collision scene. The analysis results are used to make a secondary judgment on the problem category and output a more granular secondary judgment problem category and the corresponding problem scene description. This description can be used to describe the specific situation of the scene that caused the problem.

[0221] For example, the final problem analysis results may include, but are not limited to: user emotions, frequency of problem occurrence, image keyframes, scene description information, or descriptions of the reasoning process for each piece of information.

[0222] For example, the output problem analysis results can be shown in Table 1.

[0223] Table 1

[0224] Optionally, when performing multimodal analysis, the problem analysis results can also be output by combining historical problem data. For example, based on the problem category output in step 902 above, historical data corresponding to that problem category can be queried from historical data, including the problem analysis results or feedback information of that historical problem. If the current problem has a high similarity to historical problems, the problem analysis results of the current problem can be output by referring to the data of historical problems, such as using all or part of the feedback information of historical problems as the analysis results of the current problem. Alternatively, when generating problem analysis results, the current problem can be analyzed based on the model used when analyzing historical problems, etc., depending on the actual application scenario.

[0225] Therefore, in this application, the current problem can be analyzed based on historical data, which can improve the efficiency of the analysis and output the analysis results of the current problem more efficiently.

[0226] 906. The question-and-answer system outputs and provides feedback.

[0227] After obtaining the problem breakdown results, the problem analysis results can be further organized through the problem system to output content that is easy for users or maintenance personnel to view.

[0228] For example, different content can be displayed for users and operations and maintenance personnel. For users, a more concise problem analysis result can be shown; while for operations and maintenance personnel, a more detailed problem analysis result can be shown, allowing them to further address the problem based on a more detailed problem description.

[0229] Combining steps 901 to 906 described above, the method architecture provided in this application embodiment can be simplified as shown in Figure 17.

[0230] In addition, the feedback generated includes not only a detailed description of the problem, but also corresponding solutions and possible preventative measures, such as regular maintenance recommendations, driving behavior optimization tips, and upgrade suggestions for specific systems, to help users avoid encountering similar problems again.

[0231] For example, the feedback from operations and maintenance personnel can be shown in Figure 18. To make it easier for operations and maintenance personnel to accurately analyze the causes of problems in the actual environment, richer problem analysis results can be displayed to them, such as problem type, problem occurrence time, key frame images, and a series of events that led to the problem.

[0232] For example, the feedback from users or car owners on the vehicle's end can be illustrated as shown in Figures 19 and 20. Specifically, this can include voice or text responses addressing the user's specific reasons, thus providing a more tailored response to the problem at hand.

[0233] For example, the content that users or car owners can provide feedback on the user terminal's interactive interface can be as shown in Figure 21. Specifically, it can include voice or text responses to the user's questions, thus making it more convenient for users to provide feedback on their issues on the terminal.

[0234] Of course, the content shown in the aforementioned interactive interface is only an example. The actual interactive interface settings and buttons can be determined according to the actual application scenario, and this application does not limit them.

[0235] In this embodiment, user voice feedback is fully utilized, combined with vehicle multimodal data, to construct an intelligent operation and maintenance system for intelligent operation and maintenance analysis. The system primarily uses user voice feedback as the core for problem classification and analysis, supplemented by multimodal data such as vehicle images and control signals, to achieve high precision and recall in problem identification and analysis. Finally, a feedback report is generated for use by R&D personnel or users, forming a voice-driven intelligent operation and maintenance system. This not only improves the speed of information processing but also significantly reduces the need for manual intervention. Through deep fusion of multimodal information, the system can analyze problems more comprehensively, thereby more accurately identifying and locating vehicle faults. Furthermore, the feedback reports generated by the system can be provided to the R&D team in real time to improve the product. Simultaneously, automatically generated feedback can directly interact with users, providing immediate emotional reassurance and problem feedback. This two-way feedback mechanism makes the system perform exceptionally well in problem solving and user experience improvement.

[0236] Therefore, the embodiments of this application can improve the intelligent operation and maintenance efficiency of intelligent driving systems and reduce the workload of manual maintenance. It can lower the interaction skill threshold for operation and maintenance personnel, providing more accurate problem descriptions through automated system analysis. This allows operation and maintenance personnel to prioritize issues based on user sentiment, improving the efficiency and accuracy of operation and maintenance. It also enhances the accuracy of problem verification, especially in complex problem scenarios. Furthermore, it uncovers challenging case data from intelligent driving models, assisting in improving the overall performance of the models. Finally, it improves user experience; through an intelligent feedback mechanism, users can receive faster responses and solutions to problems. Moreover, when responding to users, comprehensive sentiment analysis allows the system to positively guide user emotions, thereby reducing user anxiety to some extent.

[0237] The methods and operation and maintenance systems provided in this application can be deployed not only in the aforementioned intelligent driving scenarios, but also in other operation and maintenance scenarios. For example, the methods and operation and maintenance systems provided in this application can also be deployed in smart home systems, health assistance systems, manufacturing plant management, intelligent agricultural management, or intelligent urban maintenance, etc.

[0238] For example, in a smart home scenario, users can provide voice feedback on the usage and problems of their home devices. The system then uses multimodal data (such as from cameras and sensors) for automated analysis and troubleshooting. By combining this data with the devices' operational status, the system can identify faults and provide specific repair guidance, thereby improving the user's home experience.

[0239] For example, in health assistance systems, patients describe their symptoms via voice, and the system combines this with biosignals (such as heart rate and blood pressure) for a comprehensive diagnosis, providing medical personnel with supplementary information. The system can also use emotion analysis to assess the patient's psychological state, providing medical personnel with even more auxiliary information.

[0240] For example, in manufacturing management scenarios, factory equipment can use voice feedback to report its status, and combined with equipment operating data, it can perform automatic diagnosis and maintenance, improving the efficiency and accuracy of equipment operation and maintenance. This system can automatically identify potential equipment faults and generate maintenance suggestions, thereby reducing downtime and increasing production efficiency.

[0241] For example, in the context of intelligent agricultural management, users can provide feedback via voice on the usage of farm equipment or the growth status of crops. The system combines agricultural sensor data (such as soil moisture, temperature, and meteorological data) for comprehensive analysis and provides precise agricultural management suggestions, such as irrigation time and fertilizer application.

[0242] For example, in intelligent urban management systems, users can report problems with urban infrastructure (such as broken streetlights, potholes, etc.) via voice. The system combines data from urban IoT sensors to intelligently classify and prioritize the problems, generate maintenance suggestions, and notify relevant departments to improve the response speed and efficiency of urban management.

[0243] The methods provided in the embodiments of this application have been described above. The system and apparatus structure for performing the above methods are described below.

[0244] Referring to Figure 22, a schematic diagram of the structure of an operation and maintenance system according to this application includes: vehicle terminal 2201 and cloud terminal 2202;

[0245] The cloud-based 2202 is used to take input data as input to a large language model and output the problem category. The input data indicates how to solve the vehicle-side 2202 problem.

[0246] The cloud-based 2202 is also used to acquire multimodal data, which includes data collected by at least one sensor and data related to vehicle issues.

[0247] The cloud-based 2202 is also used to analyze problems based on problem categories and multimodal data to obtain problem analysis results;

[0248] Cloud 2202 is also used to output problem analysis results.

[0249] In one possible implementation, the aforementioned vehicle terminal 2201 is also used to display the problem analysis results.

[0250] In one possible implementation, the aforementioned system further includes a user terminal 2203 for displaying the problem analysis results.

[0251] In one possible implementation, the aforementioned user terminal 2203 is further configured to acquire input data, which is obtained based on input operations performed by the user on the user terminal's interactive interface.

[0252] In one possible implementation, the aforementioned vehicle terminal 2201 is also used to acquire input data, which is obtained based on the user's input operation on the vehicle terminal's interactive interface.

[0253] In one possible implementation, the aforementioned cloud 2202 is specifically used to: take the input data as input to a large language model, output question category and emotion information, the emotion information including the user emotion corresponding to the input data, the input data including voice data; and output question analysis results and emotion information.

[0254] In one possible implementation, the aforementioned large language model is used to filter out question categories from multiple categories in the pattern library.

[0255] In one possible implementation, the aforementioned pattern library also includes data rules corresponding to each category, wherein the data rules are data types corresponding to each category;

[0256] The aforementioned cloud-based 2202 is also used to: determine the first data rule corresponding to the problem category based on the pattern library; and obtain multimodal data from the vehicle based on the first data rule, wherein the multimodal data includes data of the type indicated by the first data rule.

[0257] In one possible implementation, the aforementioned pattern library also includes a model type corresponding to each category;

[0258] The aforementioned cloud-based 2202 is used to: determine the first model type corresponding to the problem category based on the pattern library; and analyze multimodal data based on the model indicated by the first model type to obtain the problem analysis results.

[0259] In one possible implementation, the aforementioned cloud 2202 is specifically used to: input the problem analysis results into the question-and-answer system, output the question answer, the question answer including the content of the problem analysis results, and the question answer being used to respond to the input data.

[0260] In one possible implementation, the aforementioned cloud 2202 is further configured to: generate a problem description based on the problem analysis results, the problem description including the problem analysis results and the analysis path, the analysis path being used to indicate the cause of the problem indicated by the input data; if the large language model also outputs sentiment information, the problem description also includes sentiment information.

[0261] In one possible implementation, the aforementioned cloud 2202 is further used to: obtain historical problem information, which includes information related to the problem corresponding to the input data; and analyze the problem information based on the problem category and multimodal data, combined with the historical problem information.

[0262] Referring to Figure 23, a schematic diagram of another operation and maintenance system provided in this application includes:

[0263] Input module 2301 is used to acquire input data, which is obtained based on data input by the user, and the input data indicates how to solve the vehicle problem;

[0264] The large model analysis module 2302 is used to take input data as input to the large language model and output the question category.

[0265] The multimodal data acquisition module 2303 is used to acquire multimodal data, which includes data collected by at least one sensor and data related to vehicle problems.

[0266] The multimodal analysis module 2304 is used to analyze multimodal data based on problem categories to obtain problem analysis results;

[0267] Output module 2305 is used to output the problem analysis results.

[0268] In one possible implementation, the aforementioned large model analysis module 2302 is used to: take the input data as the input of the large language model and output the question category and emotion information, wherein the emotion information includes the user emotion corresponding to the input data and the input data includes voice data;

[0269] The aforementioned output module 2305 is used to output the problem analysis results and sentiment information.

[0270] In one possible implementation, the aforementioned large language model is used to filter out question categories from multiple categories in the pattern library.

[0271] In one possible implementation, the aforementioned pattern library also includes data rules corresponding to each category, and the data rules are data types corresponding to each category; the aforementioned multimodal data acquisition module 2303 is used to: determine the first data rule corresponding to the problem category according to the pattern library; acquire multimodal data according to the first data rule, wherein the multimodal data includes data of the type indicated by the first data rule.

[0272] In one possible implementation, the aforementioned pattern library also includes a model type corresponding to each category; the aforementioned multimodal analysis module 2303 is used to: determine the first model type corresponding to the problem category according to the pattern library; and analyze the multimodal data according to the model indicated by the first model type to obtain the problem analysis result.

[0273] In one possible implementation, the aforementioned output module 2305 is used to: input the problem analysis results into the question-and-answer system, output the question answer, the question answer including the content of the problem analysis results, and the question answer being used to respond to the input data.

[0274] In one possible implementation, the aforementioned output module 2305 is further configured to: generate a problem description based on the problem analysis results, wherein the problem description includes the problem analysis results and the analysis path, and the analysis path is used to indicate the cause of the problem indicated by the input data; if the large language model also outputs sentiment information, the problem description also includes sentiment information.

[0275] In one possible implementation, the aforementioned system further includes:

[0276] The history module 2306 is used to obtain historical problem information, which includes information related to the problem corresponding to the input data.

[0277] The multimodal analysis module 2303 is used to analyze the problem information based on the problem category and multimodal data, combined with historical problem information.

[0278] Figure 24 shows a schematic diagram of the hardware structure of a computing device 240 provided in an embodiment of this application. This computing device 240 can be used to implement the steps of the methods shown in Figures 3 to 21, and may specifically include the aforementioned vehicle-side, cloud-side, or user terminal, etc.

[0279] The computing device 240 shown in Figure 24 may include a processor 2401, a memory 2402, a communication interface 2403, and a bus 2404. The processor 2401, the memory 2402, and the communication interface 2403 can be connected to each other via the bus 2404.

[0280] The processor 2401 is the control center of the computing device 240. It can be a general-purpose central processing unit (CPU) or other general-purpose processors. The general-purpose processor can be a microprocessor or any conventional processor, such as a GPU or NPU, and can be adapted to the actual application scenario.

[0281] As an example, processor 2401 may include one or more CPUs, and may also include other processors, such as the CPU, NPU or GPU shown in Figure 24.

[0282] The memory 2402 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0283] In one possible implementation, the memory 2402 may exist independently of the processor 2401. The memory 2402 can be connected to the processor 2401 via a bus 2404 and is used to store data, instructions, or program code. When the processor 2401 calls and executes the instructions or program code stored in the memory 2402, it can implement the methods provided in the embodiments of this application, such as the methods shown in Figures 3 to 21.

[0284] In another possible implementation, the memory 2402 can also be integrated with the processor 2401.

[0285] The communication interface 2403 is used for the computing device 240 to connect with other devices via a communication network, which may be Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. The communication interface 2403 may include a receiving unit for receiving data and a transmitting unit for transmitting data.

[0286] Bus 2404 can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus. This bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in Figure 24, but this does not indicate that there is only one bus or one type of bus.

[0287] It should be noted that the structure shown in FIG24 does not constitute a limitation on the computing device 240. In addition to the components shown in FIG24, the computing device 240 may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0288] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., including several instructions to cause a data quantization device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0289] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0290] This application also provides a computer-readable storage medium storing a program for training a model or performing inference tasks, which, when run on a computer, causes the computer to perform all or part of the steps in the methods described in the embodiments shown in Figures 3 to 21 above.

[0291] This application also provides a digital processing chip. This digital processing chip integrates circuitry for implementing the aforementioned processor or processor functions, and one or more interfaces. When the digital processing chip integrates a memory, it can perform the method steps of any one or more of the foregoing embodiments. When the digital processing chip does not integrate a memory, it can be connected to an external memory via a communication interface. The digital processing chip implements the method steps of any one or more of the foregoing embodiments based on the program code stored in the external memory.

[0292] This application also provides a computer program product comprising one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0293] The target sensing device or target sensing device provided in this application embodiment can be a chip, which includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip in the server to execute the method described in the embodiments shown in Figures 3-5 above. Optionally, the storage unit is a storage unit within the chip, such as a register or cache. The storage unit can also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, such as random access memory (RAM).

[0294] Specifically, the aforementioned processing unit or processor can be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0295] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0296] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0297] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0298] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0299] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. The term "and / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Additionally, the character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. The naming or numbering of steps in this application does not imply that the steps in the method flow must be executed in the time / logical order indicated by the naming or numbering. The execution order of the named or numbered process steps can be changed according to the technical purpose to be achieved, as long as the same or similar technical effect can be achieved. The division of modules in this application is a logical division. In actual applications, there may be other division methods. For example, multiple modules may be combined into or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the modules shown or discussed may be through some ports, and the indirect coupling or communication connection between modules may be electrical or other similar forms, which are not limited in this application. Furthermore, the modules or sub-modules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed in multiple circuit modules. Some or all of the modules can be selected to achieve the purpose of the solution in this application according to actual needs.

Claims

1. A data analysis method, characterized by, The method comprises: acquiring input data, the input data being obtained according to user input data, the input data indicating a vehicle problem; inputting the input data into a large language model to output a problem category; acquiring multi-modal data, the multi-modal data including at least one type of data collected by a sensor, and the multi-modal data including data related to the vehicle problem; analyzing the problem category and the multi-modal data to obtain a problem analysis result; and outputting the problem analysis result.

2. The method of claim 1, wherein, The inputting the input data into a large language model to output a problem category comprises: inputting the input data into a large language model to output the problem category and emotion information, the emotion information including user emotion corresponding to the input data, and the input data including voice data. The outputting the problem analysis result comprises: outputting the problem analysis result and the emotion information.

3. The method according to claim 1 or 2, characterized in that, The large language model is used to filter the problem category from a plurality of categories in a mode library.

4. The method of claim 3, wherein, The mode library further includes a data rule corresponding to each category, and the data rule is a data type corresponding to each category. The acquiring multi-modal data comprises: determining a first data rule corresponding to the problem category according to the mode library; and acquiring the multi-modal data according to the first data rule, the multi-modal data including data of a type indicated by the first data rule.

5. The method according to claim 3 or 4, characterized in that, The mode library further includes a model type corresponding to each category. The analyzing the problem category and the multi-modal data to obtain a problem analysis result comprises: determining a first model type corresponding to the problem category according to the mode library; and analyzing the multi-modal data according to a model indicated by the first model type to obtain the problem analysis result.

6. The method according to any one of claims 1-5, characterized in that, The outputting the problem analysis result comprises: inputting the problem analysis result into a question and answer system to output a problem reply, the problem reply including content of the problem analysis result, and the problem reply being used to reply to the input data; and outputting the problem reply through an interactive interface.

7. The method according to any one of claims 1 to 6, characterized in that, The outputting the problem analysis result further comprises: generating a problem description according to the problem analysis result, the problem description including the problem analysis result and an analysis path, and the analysis path being used to indicate a cause of a problem indicated by the input data; and if the large language model further outputs emotion information, the problem description further includes the emotion information.

8. The method according to any one of claims 1-7, characterized in that, The method further comprises: acquiring historical problem information, the historical problem information including information related to a problem corresponding to the input data; and The analyzing the problem category and the multi-modal data to obtain a problem analysis result further comprises: analyzing the problem category, the multi-modal data, and the historical problem information to obtain the problem analysis result.

9. An operation and maintenance system, characterized by The method comprises: a vehicle end and a cloud end; the cloud end is configured to input input data into a large language model to output a problem category, the input data indicating a vehicle problem. The cloud end is further configured to acquire multi-modal data, the multi-modal data including at least one type of sensor-acquired data, and the multi-modal data including data related to the vehicle problem; The cloud end is further configured to analyze the problem category and the multi-modal data to obtain a problem analysis result; The cloud end is further configured to output the problem analysis result.

10. The system of claim 9, wherein The vehicle end is further configured to display the problem analysis result.

11. The system of claim 9, wherein, The system further comprises: A user terminal configured to display the problem analysis result.

12. The system of claim 11, wherein The user terminal is further configured to acquire the input data, the input data being obtained according to user input operations on an interactive interface of the user terminal.

13. The system of claims 9-11, wherein The vehicle end is further configured to acquire the input data, the input data being obtained according to user input operations on an interactive interface of the vehicle end.

14. The system of claim 9, wherein, The cloud end is specifically configured to: input the input data into a large language model to output the problem category and emotion information, the emotion information including user emotion corresponding to the input data, and the input data including voice data; output the problem analysis result and the emotion information.

15. The system of any one of claims 9-14, wherein, The large language model is configured to filter the problem category from a plurality of categories in a pattern library.

16. The system of claim 15, wherein, The pattern library further includes a data rule corresponding to each category, the data rule being a data type corresponding to each category; The cloud end is further configured to: determine a first data rule corresponding to the problem category according to the pattern library; acquire the multi-modal data from the vehicle end according to the first data rule, the multi-modal data including data of a type indicated by the first data rule.

17. The system of claim 15 or 16, wherein, The pattern library further includes a model type corresponding to each category; The cloud end is configured to: determine a first model type corresponding to the problem category according to the pattern library; analyze the multi-modal data according to a model indicated by the first model type to obtain the problem analysis result.

18. The system of any of claims 9-17, wherein, The cloud end is specifically configured to: input the problem analysis result into a question and answer system to output a problem reply, the problem reply including content of the problem analysis result, and the problem reply being used to reply to the input data.

19. The system of any of claims 9-18, wherein, The cloud end is further configured to: generate a problem description according to the problem analysis result, the problem description including the problem analysis result and an analysis path, the analysis path being used to indicate a cause of a problem indicated by the input data; if the large language model further outputs emotion information, the problem description further includes the emotion information.

20. The system of any one of claims 9-19, wherein, The cloud end is further configured to: acquire historical problem information, the historical problem information including information related to a problem corresponding to the input data; analyze the problem category and the multi-modal data in combination with the historical problem information to obtain the problem information.

21. An operation and maintenance system, characterized by ​ An input module configured to obtain input data, the input data being obtained according to user input, the input data indicating a vehicle problem; A large model analysis module configured to input the input data into a large language model to output a problem category; A multi-modal data obtaining module configured to obtain multi-modal data, the multi-modal data including at least one type of data collected by a sensor, and the multi-modal data including data related to the vehicle problem; A multi-modal analysis module configured to analyze the problem category and the multi-modal data to obtain a problem analysis result; An output module configured to output the problem analysis result.

22. The system of claim 21, wherein, The large model analysis module is configured to: input the input data into the large language model to output the problem category and emotion information, the emotion information including user emotion corresponding to the input data, and the input data including voice data; The output module is configured to output the problem analysis result and the emotion information.

23. The system of claim 21 or 22, wherein, The large language model is configured to filter the problem category from a plurality of categories in a pattern library.

24. The system of claim 23, wherein, The pattern library further includes a data rule corresponding to each category, and the data rule is a data type corresponding to each category; The multi-modal data obtaining module is configured to: determine a first data rule corresponding to the problem category according to the pattern library; obtain the multi-modal data according to the first data rule, and the multi-modal data includes data of a type indicated by the first data rule.

25. The system of claim 23 or 24, wherein, The pattern library further includes a model type corresponding to each category; The multi-modal analysis module is configured to: determine a first model type corresponding to the problem category according to the pattern library; analyze the multi-modal data according to a model indicated by the first model type to obtain the problem analysis result.

26. The system of any of claims 21-25, wherein, The output module is configured to: input the problem analysis result into a question and answer system to output a problem reply, the problem reply including content of the problem analysis result, and the problem reply being used to reply to the input data.

27. The system of any of claims 21-26, wherein, The output module is further configured to: generate a problem description according to the problem analysis result, the problem description including the problem analysis result and an analysis path, and the analysis path being used to indicate a cause of a problem indicated by the input data; If the large language model further outputs emotion information, the problem description further includes the emotion information.

28. The system of any of claims 21-27, wherein, The system further includes: a history module configured to obtain historical problem information, the historical problem information including information related to a problem corresponding to the input data; The multi-modal analysis module is configured to analyze the problem category and the multi-modal data in combination with the historical problem information to obtain the problem information.

29. A vehicle characterized by An apparatus including a memory and a processor, the memory storing code, and the processor configured to execute the code, when the code is executed, the apparatus performing the method of any one of claims 1-8.

30. A cloud server, characterized by An apparatus comprising a memory and a processor; the memory storing code, the processor configured to execute the code, when executed, to perform the method of any of claims 1-8.

31. A vehicle characterized by An apparatus comprising a memory and a processor; the memory storing code, the processor configured to execute the code, when executed, to perform the method of any of claims 1-8.

32. A computer storage medium, comprising, A computer storage medium storing instructions that, when executed by a computer, cause the computer to perform the method of any of claims 1-8.

33. A computer program product, characterised in that, A computer program product storing instructions that, when executed by a computer, cause the computer to perform the method of any of claims 1-8.

Images