Control method and apparatus

Abstract

The present application provides a control method and an apparatus. The method comprises: acquiring a first input from a user, wherein the first input comprises a description about an object referred to by a user; on the basis of the first input and a knowledge base, determining that the object referred to by the user is a first component; and controlling a vehicle to execute an operation related to the first component, wherein the knowledge base comprises spatial position information, the spatial position information comprises at least one of orientation information of one or more components relative to the vehicle, orientation information of the first component relative to a second component, and orientation information of the first component relative to the user, the one or more components comprise the first component, and the vehicle comprises the one or more components. The solution of the present application can achieve adaptation of vehicle components to differentiated vehicle models, thereby improving the efficiency and accuracy of user intent recognition by a vehicle, and thus enhancing user experience.

Description

A control method and apparatus Technical Field

[0001] This application relates to the field of human-computer interaction, and more specifically, to a control method and apparatus. Background Technology

[0002] With the rapid iteration and updates of smart cars and the increasing trend of functional homogenization, manufacturers are adding more and more components in their vehicles in pursuit of brand differentiation. The names of these components are also becoming more and more complex, such as light field screens and window sunshades.

[0003] However, the proprietary and specialized names of in-vehicle components significantly raise the cognitive barrier for users. Since users are often unfamiliar with the official names of these components, they frequently mispronounce or fail to use voice commands when controlling devices. These issues severely impact the efficiency of user-vehicle interaction, resulting in a poor user experience. Summary of the Invention

[0004] This application provides a control method and apparatus that helps to solve the problems of mismatch between user intent and device caused by vehicle model differences during user-device interaction, as well as the mismatch between vague user expressions and precise pre-set device commands, thereby improving user experience.

[0005] In a first aspect, a control method is provided, the method comprising: acquiring a first input from a user, the first input including a description of an object pointed to by the user; determining, based on the first input and a knowledge base, that the object pointed to by the user is a first component; and controlling a vehicle to perform an operation related to the first component, wherein the knowledge base includes spatial location information, the spatial location information including at least one of the following: orientation information of one or more components relative to the vehicle, orientation information of the first component relative to a second component, and orientation information of the first component relative to the user; the one or more components include the first component, and the vehicle includes one or more components.

[0006] Based on the above technical solution, the spatial location information in the knowledge base can be used to match vehicle components with different vehicle models. By combining the user's description of the vehicle's components or functions with the information in the knowledge base, the first component that the user is pointing to can be quickly and accurately determined. This helps to improve the vehicle's efficiency and accuracy in recognizing the user's intent, thereby enhancing the user experience.

[0007] In conjunction with the first aspect, in some implementations of the first aspect, the first input includes first location information, and determining the object pointed to by the user as the first component based on the first input and the knowledge base includes: determining the first component based on the first location information and the knowledge base.

[0008] Based on the above technical solution, when the user's description includes first location information related to the first component, the first component can be quickly and accurately determined by matching the spatial location information in the knowledge base with the first location information described by the user. This is beneficial to improving the vehicle's efficiency and accuracy in recognizing user intent, thereby enhancing the user experience.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, the first input is voice input, and the object pointed to by the user is determined as the first component based on the first location information and the knowledge base, including: obtaining the text content corresponding to the voice input; and determining the first location information based on the text content.

[0010] Based on the above technical solution, users can control the vehicle via voice. The vehicle determines the user's intent by converting the voice input into text content, making it more convenient and faster for users to control the components, thereby improving the user experience.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, the knowledge base further includes at least one of the name information, function information, color information, shape information, and material information of the first component. The name information of the first component includes the name of the first component and one or more names that match the name of the first component. The method further includes: determining that the second information included in the first input matches at least one of the name information, function information, color information, shape information, and material information of the first component. The second information is a description by the user of at least one dimension of the name, function, color, shape, and material of the first component.

[0012] Based on the above technical solution, the knowledge base can cover a variety of information related to the first component. By combining the user's various descriptions of the vehicle's components or functions with the information in the knowledge base, the first component can be quickly and accurately identified. This is beneficial to further improve the vehicle's efficiency and accuracy in recognizing user intentions, thereby enhancing the user experience.

[0013] In conjunction with the first aspect, in some implementations of the first aspect, the first input includes first location information and second information; determining the object pointed to by the user as the first component based on the first input and the knowledge base includes: determining the first component based on the first location information, the second information and the knowledge base.

[0014] Based on the above technical solution, when a user describes the first location information and the second information related to the first component, the user's description can be matched with the existing spatial location information, name information, function information, color information, shape information, material information, etc. in the knowledge base. This can quickly and accurately determine the first component, which is conducive to further improving the vehicle's recognition efficiency and accuracy of user intent, thereby enhancing the user experience.

[0015] In conjunction with the first aspect, in one possible implementation, the first input includes second information; determining the object pointed to by the user as the first component based on the first input and the knowledge base includes: determining the first component based on the second information and the knowledge base.

[0016] In conjunction with the first aspect, in some implementations of the first aspect, the first input includes user information and first location information; determining the object pointed to by the user as the first component based on the first input and the knowledge base includes: determining the first area where the user is located in the vehicle's cabin based on the user information; and determining the first component based on the first area, the first location information, and the location information of the first component relative to the user.

[0017] Based on the above technical solution, the user's location can be determined according to the user's information. Then, by combining the user's location, first-direction information, and spatial location information in the knowledge base, the first component can be determined, which further improves the vehicle's efficiency and accuracy in recognizing the user's intent, thereby enhancing the user experience.

[0018] In conjunction with the first aspect, in some implementations of the first aspect, the first input includes information about the second component and first orientation information; determining the object pointed to by the user as the first component based on the first input and the knowledge base includes: determining the first component based on information about the second component, first orientation information, and orientation information of the first component relative to the second component.

[0019] Based on the above technical solution, the first component can be determined according to the relative positional relationship between components in the user's description and the spatial positional information in the knowledge base, thereby further improving the vehicle's efficiency and accuracy in recognizing user intentions and thus enhancing the user experience.

[0020] In conjunction with the first aspect, in some implementations of the first aspect, the first input includes vehicle information and first orientation information; determining the object pointed to by the user as the first component based on the first input and the knowledge base includes: determining the first component based on the vehicle information, the first orientation information and the orientation information of the first component relative to the vehicle.

[0021] Based on the above technical solution, the first component can be determined according to the relative positional relationship between the component and the vehicle in the user's description and the spatial positional information in the knowledge base, which further improves the vehicle's efficiency and accuracy in recognizing user intentions, thereby enhancing the user experience.

[0022] In conjunction with the first aspect, in some implementations of the first aspect, determining the object pointed to by the user as the first component based on the first input and the knowledge base includes: determining multiple components based on the first input and the knowledge base, wherein the multiple components include the first component; and determining the first component from the multiple components based on at least one of the user's gaze direction, gesture input, and head posture when the first input is obtained.

[0023] Based on the above technical solution, when there are multiple components that match the user's description, the first component can be determined by combining the user's gaze direction, gesture input, head posture and other action indicators, which can further improve the vehicle's efficiency and accuracy in recognizing the user's intent, thereby improving the user experience.

[0024] In conjunction with the first aspect, in some implementations of the first aspect, determining the object pointed to by the user as the first component based on the first input and the knowledge base includes: determining multiple components based on the first input and the knowledge base, wherein the multiple components include the first component; controlling the vehicle to prompt the user to select at least one of the multiple components; and determining the first component based on the user's selection result.

[0025] Based on the above technical solution, when there are multiple components that match the user's description, prompts can be used to determine the component the user actually wants to select. This avoids scenarios where the vehicle is controlled against the user's intentions due to the inability to accurately determine the user's intent, which helps improve the vehicle's efficiency and accuracy in recognizing user intents, thereby enhancing the user experience.

[0026] In conjunction with the first aspect, in some implementations of the first aspect, before determining the first component based on the user's selection, the method further includes: according to a knowledge base, controlling the vehicle to prompt the user with difference information between multiple components, the difference information including difference information of the multiple components in at least one dimension of spatial location information, functional information, color information, shape information, and material information.

[0027] Based on the above technical solution, when there are multiple components that match the user's description, the user can be prompted with additional difference information. This helps the user understand the differences between the multiple components, provides more reference information for the user's selection, and facilitates the user's choice among multiple components, thereby improving the user experience.

[0028] In conjunction with the first aspect, in some implementations of the first aspect, determining the object pointed to by the user as the first component based on the first input and the knowledge base includes: when no component is matched based on the first input and the knowledge base, controlling the vehicle to prompt the user with at least one component; and determining the first component based on the user's selection result of at least one component.

[0029] Based on the above technical solution, when it is difficult to determine the component that matches the user's description, prompts can be used to determine the component that the user actually wants to select. This avoids scenarios where the vehicle cannot be controlled due to the inability to accurately determine the user's intent, which helps to improve the vehicle's efficiency and accuracy in recognizing the user's intent, thereby enhancing the user experience.

[0030] In conjunction with the first aspect, in some implementations of the first aspect, the knowledge base includes a first knowledge base, which is a knowledge base for the vehicle. The method further includes: establishing a first association relationship between a first semantic in a first input and a first component in the first knowledge base; obtaining a second input from the user, the second input including the first semantic; and determining the first component based on the second input and the first knowledge base.

[0031] For example, the first knowledge base may be a personalized knowledge sub-base corresponding to the vehicle.

[0032] Based on the above technical solution, an association can be established in the first knowledge base according to the user's description and matching results. When the same description of the first component is obtained for the second time by the user, the first component can be directly determined based on the association. This reduces the interaction process when the user inputs input for the second time, which is conducive to further improving the vehicle's recognition efficiency of the user's intent and thus improving the user experience.

[0033] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: receiving a second knowledge base for a first region sent by a cloud server, the second knowledge base including the association relationship between the dialect information of the first semantic in the first region and the first component; obtaining a third input from the user, the third input including dialect information; and determining the first component based on the dialect information and the second knowledge base.

[0034] For example, the second knowledge base could be a regional knowledge base for the first region.

[0035] Based on the above technical solution, the knowledge base can also include the user's description of the first component in dialect and the user's regional information. The knowledge base contains richer data, which is conducive to further improving the vehicle's recognition efficiency of user intentions, thereby improving the user experience.

[0036] In a second aspect, a control device is provided, comprising: an acquisition unit for acquiring a first input from a user, the first input including a description of an object pointed to by the user; a determination unit for determining, based on the first input and a knowledge base, that the object pointed to by the user is a first component; and a control unit for controlling a vehicle to perform an operation related to the first component, wherein the knowledge base includes spatial location information, the spatial location information including at least one of the following: the orientation information of one or more components relative to the vehicle, the orientation information of the first component relative to a second component, and the orientation information of the first component relative to the user; the one or more components include the first component; and the vehicle includes one or more components.

[0037] In conjunction with the second aspect, in some implementations of the second aspect, the first input includes first orientation information, and the determining unit is specifically used to: determine the first component based on the first orientation information and a knowledge base.

[0038] In conjunction with the second aspect, in some implementations of the second aspect, the first input is voice input, and the determining unit is specifically used for: acquiring the text content corresponding to the voice input; and determining the first location information based on the text content.

[0039] In conjunction with the second aspect, in some implementations of the second aspect, the knowledge base further includes at least one of the name information, function information, color information, shape information, and material information of the first component. The name information of the first component includes the name of the first component and one or more names that match the name of the first component. The determining unit is further configured to: determine that the second information included in the first input matches at least one of the name information, function information, color information, shape information, and material information of the first component. The second information is a description by the user of at least one dimension of the name, function, color, shape, and material of the first component.

[0040] In conjunction with the second aspect, in some implementations of the second aspect, the first input includes first orientation information and second information; the determining unit is specifically used to: determine the first component based on the first orientation information, the second information, and the knowledge base.

[0041] In conjunction with the second aspect, in one possible implementation, the first input includes second information; the determining unit is specifically used to: determine the first component based on the second information and the knowledge base.

[0042] In conjunction with the second aspect, in some implementations of the second aspect, the first input includes user information and first orientation information; the determining unit is specifically used to: determine the first area where the user is located in the vehicle cabin based on the user information; and determine the first component based on the first area, the first orientation information, and the orientation information of the first component relative to the user.

[0043] In conjunction with the second aspect, in some implementations of the second aspect, the first input includes information about the second component and first orientation information; the determining unit is specifically used to: determine the first component based on the information about the second component, the first orientation information, and the orientation information of the first component relative to the second component.

[0044] In conjunction with the second aspect, in some implementations of the second aspect, the first input includes vehicle information and first orientation information; the determining unit is specifically used to: determine the first component based on the vehicle information, the first orientation information and the orientation information of the first component relative to the vehicle.

[0045] In conjunction with the second aspect, in some implementations of the second aspect, the determining unit is specifically used for: determining multiple components, including the first component, based on the first input and a knowledge base; and determining the first component from the multiple components based on at least one of the user's gaze direction, gesture input, and head posture when the first input is obtained.

[0046] In conjunction with the second aspect, in some implementations of the second aspect, the determining unit is specifically used to: determine multiple components based on the first input and the knowledge base, the multiple components including the first component; the control unit is further used to control the vehicle to prompt the user to select at least one of the multiple components; the determining unit is specifically used to: determine the first component based on the user's selection result.

[0047] In conjunction with the second aspect, in some implementations of the second aspect, before determining the first component based on the user's selection, the control unit is further configured to: based on a knowledge base, control the vehicle to prompt the user with difference information between multiple components, the difference information including differences in at least one dimension of the multiple components in terms of spatial location information, functional information, color information, shape information, and material information.

[0048] In conjunction with the second aspect, in some implementations of the second aspect, the control unit is further configured to: when no component is matched based on the first input and the knowledge base, control the vehicle to prompt the user for at least one component; the determining unit is specifically configured to: determine the first component based on the user's selection result for at least one component.

[0049] In conjunction with the second aspect, in some implementations of the second aspect, the knowledge base includes a first knowledge base, which is a knowledge base for the vehicle. The device further includes: an establishment unit, configured to establish a first association relationship between a first semantic in a first input and a first component in the first knowledge base; an acquisition unit, further configured to acquire a second input from the user, the second input including the first semantic; and a determination unit, further configured to determine the first component based on the second input and the first knowledge base.

[0050] In conjunction with the second aspect, in some implementations of the second aspect, the knowledge base includes a second knowledge base, and the device further includes: a receiving unit for receiving a second knowledge base for a first region sent by a cloud server, the second knowledge base including the association relationship between dialect information of the first semantic in the first region and the first component; an acquiring unit for acquiring a third input from the user, the third input including dialect information; and a determining unit for determining the first component based on the dialect information and the second knowledge base.

[0051] Thirdly, a control device is provided, comprising: a memory for storing a computer program; and a processor for executing the computer program stored in the memory, such that the device performs a method corresponding to any of the implementations of the first aspect above.

[0052] Fourthly, a vehicle is provided, including a device corresponding to any of the implementations of the second or third aspect above.

[0053] Fifthly, a computer-readable storage medium is provided having instructions stored thereon that, when executed by a processor, cause the processor to implement a method corresponding to any of the implementations in the first aspect above.

[0054] Sixthly, a computer program product is provided, comprising computer program code, which, when run on a computer, enables the computer to implement a method corresponding to any of the implementation methods in the first aspect above.

[0055] In a seventh aspect, a chip is provided, the chip including circuitry, the circuitry being used to execute a method corresponding to any of the implementations in the first aspect above. Attached Figure Description

[0056] Figure 1 is a functional schematic diagram of a vehicle 100 provided in an embodiment of this application.

[0057] Figure 2 is a schematic diagram of a knowledge base construction and use according to an embodiment of this application.

[0058] Figure 3 is a schematic flowchart of a control method 300 according to an embodiment of this application.

[0059] Figure 4 is an example diagram of a graphical user interface (GUI) provided in an embodiment of this application.

[0060] Figure 5 is an example diagram of a GUI provided in an embodiment of this application.

[0061] Figure 6 is a schematic diagram of the distribution of vehicle components according to an embodiment of this application.

[0062] Figure 7 is a schematic flowchart of a knowledge base construction method 700 according to an embodiment of this application.

[0063] Figure 8 is a schematic flowchart of a control method 800 according to an embodiment of this application.

[0064] Figure 9 is a schematic diagram of an online knowledge base learning scheme according to an embodiment of this application.

[0065] Figure 10 is a schematic diagram of a knowledge base update method 1000 according to an embodiment of this application.

[0066] Figure 11 is a schematic block diagram of a control device 1100 provided in an embodiment of this application. Detailed Implementation

[0067] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. "At least one" refers to one or more. For example, "at least one of A and B," similar to "A and / or B," describes the association relationship between related objects, indicating that three relationships can exist. For example, at least one of A and B can represent: A existing alone, A and B existing simultaneously, and B existing alone.

[0068] The prefixes such as "first" and "second" used in this application embodiment are merely for distinguishing different descriptive objects and do not limit the position, order, priority, quantity, or content of the described objects. The use of ordinal numbers and other prefixes used to distinguish descriptive objects in this application embodiment does not constitute a limitation on the described objects. The description of the described objects is given in the claims or the context of the embodiments, and should not constitute unnecessary restrictions due to the use of such prefixes. Furthermore, in the description of this embodiment, unless otherwise stated, "multiple" means two or more.

[0069] Figure 1 is a functional schematic diagram of a vehicle 100 provided in an embodiment of this application. The vehicle 100 may include a sensing system 110, a computing platform 120, and a display device 130. The sensing system 110 may include one or more sensors and camera devices for sensing information about the environment surrounding and inside the vehicle 100. For example, the sensing system 110 may include a positioning system, which may be a Global Positioning System (GPS), a BeiDou system, or another positioning system. As another example, the sensing system 110 may include one or more of an inertial measurement unit (IMU), an accelerometer, a lidar, millimeter-wave radar, ultrasonic radar, and camera devices. Additionally, the sensing system 110 may also include a voice system, which may include one or more microphones and speakers, and can be set to different locations within the vehicle 100.

[0070] Some or all of the functions of vehicle 100 can be controlled by computing platform 120. Computing platform 120 may include one or more processors, such as processors 121 to 12n (n being a positive integer). A processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction read and execute capabilities, such as a central processing unit (CPU), microprocessor, graphics processing unit (GPU) (which can be understood as a type of microprocessor), or digital signal processor (DSP). In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. These logical relationships are fixed or reconfigurable. For example, the processor may be a hardware circuit implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as a field-programmable gate array (FPGA). In reconfigurable hardware circuits, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the process of the processor loading instructions to implement some or all of the functions of the aforementioned units. Furthermore, the processor can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), tensor processing unit (TPU), deep learning processing unit (DPU), etc. In addition, the computing platform 120 may also include a memory for storing instructions. Some or all of the processors 121 to 12n can call the instructions in the memory to implement the corresponding functions.

[0071] The display devices 130 within the vehicle's cabin are mainly divided into two categories: the first is in-vehicle displays; the second is projection displays, such as head-up displays (HUDs). In-vehicle displays are physical displays and an important component of the in-vehicle infotainment system. Multiple displays can be installed in the cabin, such as digital instrument cluster displays, central control screens, displays in front of the front passenger (also known as the front-seat user), displays in front of the left and right rear-seat users, and even the windows can be used as displays. Head-up displays, also known as head-up display systems, are mainly used to display driving information such as speed and navigation on a display device in front of the driver (e.g., the windshield). This reduces driver eye movement time, avoids pupil changes caused by eye movement, and improves driving safety and comfort. Examples of HUDs include combiner-HUD (C-HUD) systems, windshield-HUD (W-HUD) systems, and augmented reality HUD (AR-HUD) systems. It should be understood that HUDs can also evolve into other types of systems as technology progresses, and the embodiments of this application do not limit this.

[0072] The above description of the display device 130 uses an in-vehicle display screen and a projection display screen as examples, but the embodiments of this application are not limited thereto. For example, the display device 130 can also be a light display screen or a projection screen.

[0073] Optionally, the structure of the vehicle 100 described above is merely illustrative. In actual applications, various components of the vehicle 100 may be added or removed as needed.

[0074] With the rapid iteration and increasing homogenization of functions in smart cars, manufacturers are producing more and more vehicle components in pursuit of brand differentiation, and the names of these components are becoming increasingly complex. Users are often unaware of the official names of these components, leading to frequent instances of users being unable to speak correctly or accurately when controlling in-car devices via voice commands. For example, existing vehicles are equipped with various lighting devices both inside and outside. Users might say things like, "Turn off the rear lights," "Turn on the interior soft lights," "Turn off the orange lights," "Turn on the running lights," or "Turn off the front lights." Since the names of the lights in these statements are not the official names of the interior lights, the vehicle "doesn't understand" them, resulting in the vehicle being unable to respond to the user's voice commands or perform the corresponding operations. Furthermore, existing human-machine interaction solutions may have incompatibility issues when applied to different vehicle models. For example, when a user issues the command "unlock the top window," for vehicle 1, which is only equipped with an open electric sunshade (hereinafter referred to as the sunshade), vehicle 1 will open the sunshade. However, for vehicle 2, if it is equipped with a closed electric sunroof (hereinafter referred to as the sunroof) and an open sunshade, in response to the command "unlock the top window," vehicle 2 may open the sunroof. This solution causes some confusion for users controlling the vehicle, making it difficult to accurately respond to their control needs and resulting in a poor user experience.

[0075] To address these issues, this application provides a control method and apparatus that, by constructing a platform-based knowledge base, resolves the problems of mismatched user intent and mismatch between vague user expressions and precise pre-set instructions caused by device differences during user-device interaction.

[0076] The following uses vehicle 100 as an example to introduce the knowledge base construction method and device control method of this application embodiment. Figure 2 shows a schematic diagram of knowledge base construction and use according to an embodiment of this application. As shown in Figure 2, the scheme includes two parts: knowledge base construction and device control process. In the knowledge base construction part, by acquiring information such as 3D car models or whole vehicle design prototypes, user manuals, configuration files, hypernym dictionaries, and equipment production materials related to the vehicle, this information is further calibrated and integrated to obtain information related to the vehicle, including equipment spatial location, name, function, color, shape, and material. Based on this information, a knowledge base is formed. This knowledge base can be a general knowledge base, a regional knowledge base corresponding to the vehicle, or a personalized knowledge sub-base, and can be stored in the cloud or on the vehicle. Users can input information to the vehicle through voice, text, or multimodal reference. After receiving the user's input, the vehicle performs a knowledge query. By matching the user's input with the information stored in the knowledge base, the user's true intention is determined, thereby controlling the vehicle to perform device-related operations corresponding to the user's input. Among them, multimodal reference refers to the method of replacing some locative words with gesture input, gaze direction, head posture, etc. System output includes device command responses or interactive feedback. Commands refer to command signals or combinations of signals that enable the system to precisely control changes in the state of devices. Interactive feedback refers to information such as text, broadcasts, images, or combinations thereof that the system provides to the user.

[0077] Figure 3 shows a schematic flowchart of a control method 300 according to an embodiment of this application. The method 300 can be executed by the vehicle 100 or a server, or by the computing platform 120, or by a chip-on-system (SoC) in the computing platform 120, or by a processor in the computing platform 120. As shown in Figure 3, the method 300 includes:

[0078] S310: Obtain first input from the user, which includes a description of the object the user is pointing to.

[0079] For example, the first input represents the user's intent, and the object the user points to can be a vehicle component or function that the user wants to describe, inquire about, or control. The first input can be voice input or text input, and can also be combined with gesture input, gaze direction, head posture, and other actions. The first input can be a command, such as "open the sunshade," or a question, such as "what are the color-changing lights in the car called?"

[0080] For example, gesture input can be finger pointing, hand movements, or hand positions; head posture can be the user's facial orientation, head movements (such as nodding, turning, or tilting the head); and gaze direction can refer to the direction the user's eyes are looking. Multiple cameras can be installed in different locations within the vehicle cabin. When the first input includes gesture input, gaze direction, head posture, etc., the user's actions are recognized through these cameras to obtain the user's intent.

[0081] Optionally, the method 300 may further include: acquiring the status information of the first component. The status information of the first component may include function information such as on / off status, setting status, or usage mode. The component status may include function information such as on / off status, setting status, or usage mode. For example, for an ambient light component, the component status may be flashing, constantly lit, color changing, or various special status modes, such as night mode, warm mode, music-linked mode, etc.

[0082] S320: Based on the first input and the knowledge base, determine that the object pointed to by the user is the first component.

[0083] S330: Control the vehicle to perform an operation related to the first component, wherein the knowledge base includes spatial location information, which includes at least one of the following: the orientation information of one or more components relative to the vehicle, the orientation information of the first component relative to the second component, and the orientation information of the first component relative to the user; one or more components include the first component, and the vehicle includes one or more components.

[0084] For example, the first component may be a vehicle component or function that the user wants to inquire about or control, as determined by the user's first input and the knowledge base in this application embodiment. The first component may be a device within the vehicle cabin, and the second component may also be a device within the vehicle cabin or other components. For example, other components may be accessories added by the user to the vehicle, such as a phone holder or a dashcam. The location information of one or more components relative to the vehicle may refer to vehicle body location terms, indicating the spatial semantic relationship between the components and the vehicle body; the location information of the first component relative to the second component may refer to location terms relative to the component positions, indicating the spatial semantic relationship between the components; the location information of the first component relative to the user may refer to location terms relative to the user positions, indicating the spatial semantic relationship between the component and the positions of each driver and passenger. The above spatial location information can be manually labeled or obtained automatically.

[0085] This knowledge base can be a general knowledge base, a regional knowledge base, or a personalized knowledge sub-base. The construction method of the knowledge base will be described in detail in method 400 below, and will not be elaborated on here for the sake of brevity.

[0086] For example, controlling the vehicle to perform operations related to the first component may be controlling the vehicle to execute instructions related to opening, closing, or other changes in the state of the first component, or displaying text and / or image information related to the first component using a display device, or answering questions related to the first component via voice, or prompting or asking the user again in conjunction with the first component, etc. The embodiments of this application do not limit this.

[0087] Figure 4 shows an example diagram of a GUI provided in an embodiment of this application. As shown in Figure 4, user A is located in the driver's area of ​​the vehicle cabin, and the voice input is "Open the sunshade in front of me". After obtaining the voice input, based on the voice input and the knowledge base, when it is determined that the first component is a sunshade, the vehicle can be controlled to open the sunshade. At the same time, the display screen 401 can display text content corresponding to the user's command, such as "Okay, this will open the sunshade for you".

[0088] Figure 5 shows an example diagram of a GUI provided in an embodiment of this application. User A's voice input is "What is this board above my head for?" After obtaining this voice input, based on the voice input and the knowledge base, when it is determined that the first component is a sunshade, the vehicle can be controlled to use a speaker to announce "The sunshade can block your glaring sunlight" while displaying relevant text or images on the screen. As shown in Figure 5(a), the display screen 501 can display "The sunshade can block your glaring sunlight" and images corresponding to the function of the sunshade.

[0089] Optionally, if the first input is a functional question, such as "What is this panel above my head for (user in the driver's seat)?", after answering the question about the sun visor's function, "The sun visor can block the glare of the sun," further questions can be asked of the user. As shown in Figure 5(b), the display screen can be used to further display text or images such as "Would you like me to open the sun visor?", and / or the speaker can be used to announce "Would you like me to open the sun visor?". When the user responds "Open", the vehicle can be controlled to open the sun visor.

[0090] Optionally, in the method 300, the first input may include first location information, and determining the object pointed to by the user as the first component based on the first input and the knowledge base includes: determining the first component based on the first location information and the knowledge base.

[0091] For example, the first location information can represent information related to the location of the first component, such as descriptions like "on the roof," "next to," "side," "this," "that," etc. When the user's first input includes the first location information, this first location information can be matched with spatial location information in the knowledge base to determine the first component.

[0092] Optionally, in the method 300, the first input can be voice input, and the object pointed to by the user is determined as the first component based on the first location information and the knowledge base, including: obtaining the text content corresponding to the voice input; and determining the first location information based on the text content.

[0093] In one implementation, multiple microphones at different locations can be used to receive the user's voice, convert the voice into text, and then determine the text intent. Speech-to-text conversion can be achieved using an automatic speech recognition (ASR) algorithm. Extracting information contained within the text content can be achieved by splitting keywords or rewriting a large model; these two methods will be described in detail in methods 800 and 900 below, and for the sake of brevity, will not be elaborated upon here.

[0094] Optionally, in method 300, the knowledge base may further include at least one of the name information, function information, color information, shape information, and material information of the first component. The name information of the first component includes the name of the first component and one or more names that match the name of the first component. The method further includes: determining that the second information included in the first input matches at least one of the name information, function information, color information, shape information, and material information of the first component. The second information is a description by the user of at least one dimension of the name, function, color, shape, and material of the first component.

[0095] For example, the name of the first component can be its proper name or official name, and one or more names matching the name of the first component can include superordinate terms, synonyms, or near-synonyms of the proper name of the first component. The second information can be a user's description of at least one dimension of the component's name, function, color, shape, and material, such as descriptions like "curtain," "flashing," "red," "columnar," "glass," or "black button with a red triangle."

[0096] Optionally, in the method 300, the first input may include first location information and second information; determining the object pointed to by the user as the first component based on the first input and the knowledge base includes: determining the first component based on the first location information, the second information and the knowledge base.

[0097] For example, the first location information and the second information can be descriptions such as "the box next to me," "the glass on the roof," or "the lever to the left of the steering wheel." The first component is determined by matching the first location information in the first input with spatial location information in the knowledge base, and by matching the second information with at least one of name information, function information, color information, shape information, and material information in the knowledge base.

[0098] Optionally, in the method 300, the first input may include second information; determining the object pointed to by the user as the first component based on the first input and the knowledge base includes: determining the first component based on the second information and the knowledge base.

[0099] For example, the second information could be a description such as "a flashing light," "gray cloth," or "a round button." The first component is determined by matching the second information in the first input with at least one of the following in the knowledge base: name information, function information, color information, shape information, and material information.

[0100] Optionally, in the method 300, the first input includes user information and first orientation information; determining the object pointed to by the user as the first component based on the first input and the knowledge base includes: determining the first area where the user is located in the vehicle cabin based on the user information; and determining the first component based on the first area, the first orientation information, and the orientation information of the first component relative to the user.

[0101] For example, user information may include the user's actions of voice or text input and pronouns representing the user. The first location information is also combined with pronouns related to the user's information, such as "on my left," "above my head," "this," "that," etc. The vehicle cannot directly determine the corresponding part based on this type of first location information; the first location information needs to be further converted to determine the specific referent of these pronouns.

[0102] The first area where a user is located within the vehicle's cabin can be determined using a microphone array or a camera. In one embodiment, multiple microphones within the vehicle's cabin can form a microphone array, which can be used for sound source localization. For example, when a user speaks in the cabin, the loudness of the speech received by multiple microphones within the cabin at the same time varies. Based on the principle that sound loudness decreases with increasing distance, and considering the positions of the microphones within the vehicle, the microphone closest to the user receives the loudest sound, thus determining the first area where the user is located within the vehicle's cabin. Alternatively, the microphone array can also utilize time difference of arrival (TDOA) sound source localization technology. It calculates the time difference between the arrival of a sound source at each pair of microphones, thereby obtaining a system of equations for the sound source's location coordinates. Solving these equations yields the precise azimuth coordinates of the sound source, thus determining the first area where the user is located within the vehicle's cabin.

[0103] In one implementation, upon receiving the first input, multiple cameras inside the vehicle can simultaneously capture at least one image. If the image information captured by these cameras indicates that there is only one user in the cabin, the user's location within the vehicle's cabin can be directly determined. When the user inputs text, for example, when the user types "open the window next to me" on the display screen, the system detects that the user is inputting text via the display device and controls the multiple cameras inside the vehicle to capture at least one image simultaneously with the user's input. Based on the image information captured by these cameras, it can be determined which user is inputting text, thereby determining the user's location within the vehicle's cabin's first area.

[0104] Figure 6 shows a schematic diagram of vehicle component distribution according to an embodiment of this application, where (a) in Figure 6 represents the rear seat view of the vehicle cabin, and (b) in Figure 6 represents the roof view. When the vehicle cabin is equipped with multiple displays, the location of the display can be used to indirectly determine the first area where the user is located. As shown in the display 601 in Figure 6, the display 601 is located behind the driver's seat, facing the user on the left rear seat. If it is detected that the user is inputting "open the window next to me" through the display 601, the location of the display 601 can be used to determine that the user's first area in the vehicle cabin is the left rear seat.

[0105] Optionally, in the method 300, the first input includes information about the second component and first orientation information; determining the object pointed to by the user as the first component based on the first input and the knowledge base includes: determining the first component based on information about the second component, first orientation information, and orientation information of the first component relative to the second component.

[0106] For example, the information of the second component and the first location information could be descriptions such as "on the left side of the steering wheel," "on the center console," or "on the door." This information is matched with the location information of the first component relative to the second component in the knowledge base to determine the first component.

[0107] Optionally, in the method 300, the first input includes vehicle information and first orientation information; determining the object pointed to by the user as the first component based on the first input and the knowledge base includes: determining the first component based on the vehicle information, the first orientation information and the orientation information of the first component relative to the vehicle.

[0108] For example, vehicle information may include the vehicle's name, and the vehicle information and first location information may be descriptions such as "on the roof" or "behind the vehicle." This information is matched with the location information of the first component relative to the vehicle in the knowledge base to determine the first component.

[0109] Optionally, in the method 300, determining the object pointed to by the user as the first component based on the first input and the knowledge base includes: determining multiple components, including the first component, based on the first input and the knowledge base; and determining the first component from the multiple components based on at least one of the user's gaze direction, gesture input, and head posture when the first input is obtained.

[0110] For example, when obtaining the first input, at least one of the user's gaze direction, gesture input, and head posture can represent at least one of the user's gaze direction, gesture input, and head posture before obtaining the first input, or at least one of the user's gaze direction, gesture input, and head posture during the time period of obtaining the first input, or at least one of the user's gaze direction, gesture input, and head posture during a time period after obtaining the first input.

[0111] In one possible scenario, taking the user's voice command "Open the sunshade above my head" as an example, the knowledge base determines that the components matching this first input are "sunshade curtain" and "sunshade panel". To further determine the first component from the above two components, image information captured by the camera corresponding to the user issuing the voice command can be obtained. The user may also be looking up at the sunshade panel when issuing the voice command, so based on the user's line of sight, the first component can be further determined to be the "sunshade panel".

[0112] Optionally, in the method 300, determining the object pointed to by the user as the first component based on the first input and the knowledge base includes: determining multiple components, including the first component, based on the first input and the knowledge base; controlling the vehicle to prompt the user to select at least one of the multiple components; and determining the first component based on the user's selection result.

[0113] For example, taking the user's voice command "Open the sunshade above my head" as an example, the knowledge base determines that the components matching this first input are "sunshade curtain" and "glare shield". To further determine the first component from the above two components, the vehicle can be controlled to ask the user via voice "Do you want to open the sunshade curtain or the glare shield", and / or, the display screen can be used to show the user "Do you want to open the sunshade curtain or the glare shield?" When the user answers "glare shield", the first component is determined to be the glare shield.

[0114] Optionally, in method 300, before determining the first component based on the user's selection result, the method further includes: according to a knowledge base, controlling the vehicle to prompt the user with difference information between multiple components, the difference information including difference information of multiple components in at least one dimension of spatial location information, functional information, color information, shape information, and material information.

[0115] For example, suppose the driver's voice input is "Open the box next to me." Based on the knowledge base, the components matching this first input are determined to be "glove box" and "armrest box." Based on the specific locations of these two components, the name information can be combined with spatial location information. The display device can show the text "There is a glove box in front of you, and an armrest box on your left. Which one would you like to open?" and / or the speaker can broadcast the same message. The user can answer "the one in front" or "the glove box" by typing it on the display device or by speaking. Based on the user's selection, the first component is determined to be the glove box.

[0116] Optionally, in the method 300, determining the object pointed to by the user as the first component based on the first input and the knowledge base includes: when no component is matched based on the first input and the knowledge base, controlling the vehicle to prompt the user with at least one component; and determining the first component based on the user's selection result of at least one component.

[0117] For example, suppose the driver's voice input is "Open this sunshade in front of me." If no matching component can be found in the knowledge base, the system can prompt the user based on components in the user's vicinity. For instance, if there is a sunshade above the user's head, and the component functions similarly to the user's "this sunshade," the system can display the text "You can try opening the sunshade" on a display device and / or broadcast the same message via a speaker. The user can respond with "Okay" by typing it on the display device or speaking it. Based on the user's selection, the first component is determined to be the sunshade.

[0118] Optionally, in the method 300, the knowledge base includes a first knowledge base, which is a knowledge base for the vehicle. The method further includes: establishing a first association relationship between the first semantics in the first input and the first component in the first knowledge base; obtaining a second input from the user, where the second input includes the first semantics; and determining the first component according to the second input and the first knowledge base.

[0119] Exemplarily, the first knowledge base may be a personalized knowledge sub - library, which is only applied to the interaction process between the vehicle and the user and does not affect other vehicles. The first semantics may be the user's description of the first component.

[0120] Exemplarily, assume that the first input of a vehicle user is "turn on this colored light". According to the matching result, a first association relationship between the first semantics "colored light" and the first component "ambient light" can be established, and the first association relationship <colored light, ambient light> is established in the first knowledge base corresponding to the vehicle. When the second input of the vehicle user also includes the first semantics, for example, the second input may be "turn off the colored light", the first component can be determined as the ambient light according to the second input of the vehicle user and the first association relationship stored in the first knowledge base.

[0121] Optionally, in the method 300, the method further includes: receiving a second knowledge base for the first region sent by the cloud server, where the second knowledge base includes the association relationship between the dialect information of the first semantics in the first input in the first region and the first component; obtaining a third input from the user, where the third input includes the dialect information; and determining the first component according to the dialect information and the second knowledge base.

[0122] Exemplarily, the second knowledge base may be a regional knowledge base, which can be used for all vehicles of the same model and in the same region. The regional knowledge base may include dialect information corresponding to the region, such as spatial location information, function information, color information, shape information, material information, name information, etc. related to the dialect of the region. Exemplarily, taking the spatial location information "next to" as an example, the spatial location information related to the dialect of region B1 is "bian lar"; taking the function information "flashing" as an example, the function information related to the dialect of region B2 is "jian shanshan". The establishment process of the regional knowledge base can refer to the description in the following method 1000. For the sake of brevity, it will not be elaborated here.

[0123] Exemplarily, when obtaining the first input from the user, information related to the region and dialect corresponding to the user can also be obtained. The way to obtain the dialect information can be to obtain it according to the pronunciation when the user makes a voice input, or to obtain it according to the special name of a component in a specific area. When the third input of the user includes the dialect information, the first component can be determined according to the dialect information and the information in the corresponding regional knowledge base.

[0124] Based on the methods in the above embodiments, the problems of mismatch between user intent and in-vehicle components caused by vehicle model differences and mismatch between vague user expressions and precise pre-set instructions during user-vehicle interaction are solved, thereby improving the vehicle's efficiency and accuracy in recognizing user intent and enhancing user experience.

[0125] The methods in the above embodiments rely on various information related to components in the knowledge base. The following describes a knowledge base construction method 700 according to an embodiment of this application. Figure 7 shows a schematic flowchart of a knowledge base construction method 700 according to an embodiment of this application. As shown in Figure 7, the knowledge base construction method 700 includes:

[0126] S710: Obtain first information related to the vehicle.

[0127] For example, the primary information related to the vehicle can be information directly obtainable from the vehicle manufacturer or the internet, including vehicle design prototypes, 3D car models, configuration files, hypernym dictionaries, professional thesaurus, user manuals (or product instruction manuals), etc. Configuration files refer to the detailed specification list provided by the manufacturer when the vehicle leaves the factory, which may include configuration terms, configuration tables, etc., containing various configuration information of the vehicle and its components, such as interior configuration, safety equipment, exterior design, electronic components, etc.

[0128] S720: Based on the first information, obtain second information related to the components of the vehicle.

[0129] For example, the second information related to the vehicle refers to specific information about the vehicle's interior components obtained by further calculation, calibration, or generalization of the first information, including spatial location information, name information, functional information, color information, shape information, material information, etc. The name information may include component names, superordinate terms of component names, synonyms of component names, near-synonyms of component names, etc.

[0130] In one implementation, spatial location information can be obtained by using a vehicle design prototype or a 3D vehicle model to establish a coordinate system and mark the locations of each component and the user. Semantic relationships between components in space, between components and the vehicle itself in space, and between components and the user's location in space can be generated by calculating coordinate relationships. For example, using a vehicle design prototype or a 3D vehicle model, the locations of components inside the vehicle and the user's location can be marked within the prototype or model, and these locations can be quantitatively represented using a coordinate system.

[0131] As shown in Figure 6, a coordinate system is established with the left rear of the vehicle as the origin, the direction from the rear of the vehicle to the front as the x-direction, the direction from the origin to the right side of the vehicle as the y-direction, and the direction from the bottom of the vehicle to the roof as the z-direction. Taking the in-vehicle display screen and sunroof as examples, the display screen 601 is located behind the driver's seat, and the sunroof 602 is located on the top of the rear seats (not shown in Figure 6(b)). Based on the coordinate system and the vehicle size data in the 3D car model, the coordinate positions of the display screen 601 are calculated as (x1, y1, z1), the sunroof 602 as (x2, y2, z2), and the driver as (x3, y3, z3). All these coordinate positions are calibrated according to the center of the component or the person. From the coordinate system and the positional relationship between the display screen 601, sunroof 602, and driver shown in the figure, we know that x3 > x1 > x2, y3 = y1 < y2, and z2 > z1 > z3. Based on x3 > x1 > x2, the location of the display screen 601 relative to the user (driver) can be determined as "rear side," and the location of the display screen 601 relative to the component (sunroof 602) can be determined as "front side." Based on y3 = y1 < y2, the location of the display screen 601 relative to the component (sunroof 602) can be determined as "left side." Based on z2 > z1 > z3, the location of the display screen 601 relative to the component (sunroof 602) can be determined as "below." Assuming the vehicle's length is x0, width is y0, and height is z0, by calculating z0 - z3 = 0, the location of the sunroof 602 relative to the vehicle body can be determined as "top of the vehicle."

[0132] In one implementation, component names can be obtained from user manuals or configuration files. Hypernyms of component names can be obtained from a hypernym dictionary. A hypernym dictionary can be obtained directly from the internet, open-source databases, or constructed using a text decomposition method. This text decomposition method includes: obtaining the component name; identifying identical words based on the component name; and identifying hypernyms of the component name based on these identical words. For example, a vehicle can be configured with various lights such as high beam headlights, low beam headlights, fog lights, reading lights, and ambient lights. According to the text decomposition method, the component names "high beam headlights," "low beam headlights," "fog lights," "reading lights," and "ambient lights" all contain the same word "light." Since the other parts of these component names are descriptive modifiers describing the function or state of "light," "light" is identified as an identical word in these component names, and since this identical word is a noun, "light" is determined to be a hypernym of these component names. Besides obtaining synonyms from professional thesaurus and existing open-source databases, component name synonyms can also be obtained using word embeddings similarity calculation methods in large language models (LLMs). Taking the component name "sunshade curtain" as an example, "sunshade curtain" is converted into a word vector. By calculating the cosine similarity, the k words with the highest similarity to "sunshade curtain" are obtained. These words are the synonyms of "sunshade curtain." Here, k is an integer greater than 0, and its value can be adjusted according to a preset similarity threshold. Synonyms for "sunshade curtain" obtained in this way can include "light-blocking curtain," "shade curtain," and "sunshade screen," etc.

[0133] Functional information can be obtained from the user manual or configuration files. For example, vehicle model information, version information, and accessory information can be obtained from the user manual or configuration files. For instance, a brand might produce three versions of the same model: a low-end version, a mid-range version, and a high-end version. The low-end version includes only basic vehicle functions, the mid-range version adds features like a sunroof and sunshades, and the high-end version adds features like heated seats and automatic parking. Based on the model and configuration information, the functional information of the corresponding vehicle's interior components can be accurately obtained. Shape information can be obtained from the vehicle design prototype or user manual, while color information can be obtained from configuration files, 3D vehicle models, or the vehicle design prototype. For example, after obtaining the vehicle design prototype or 3D vehicle model, multiple sets of images can be taken of different areas, and the color and shape information of the corresponding components can be obtained from the information in these images.

[0134] In one implementation, material information can be obtained through configuration files, material specifications, and component manufacturing materials. For example, the codes for configuration words in the configuration file can represent different materials; for instance, in the seat configuration word, number 1 represents Nappa leather, number 2 represents fiber, and number 3 represents linen.

[0135] S730: Construct a knowledge base based on the first and second information.

[0136] The first and second information obtained from the above steps, including vehicle configuration, type, version, spatial location information, name information, function information, color information, shape information, and material information, are combined to form a knowledge base.

[0137] For example, to facilitate the classification and retrieval of information in the knowledge base, vehicle configuration, type, version, spatial location information, name information, functional information, color information, shape information, and material information can be tagged. Under each tag, there is specific knowledge related to vehicle components. For instance, the tag "name information" includes component names or superordinate terms such as door, steering wheel, seat, light, sunroof, and button; the tag "material information" includes specific material information such as plastic, cloth, glass, and leather.

[0138] For example, the knowledge base can be stored either on the vehicle or in the cloud.

[0139] The above knowledge base can be referred to as the master knowledge base. In one implementation, this master knowledge base can also be customized to meet user needs, adding user-related information to the content of the master knowledge base to form a personalized knowledge sub-base. For example, if the user of vehicle C does not like the name "ambient light," they can change the name to "colorful light" through input on the display device or voice control. The component name "colorful light" is then stored in the personalized knowledge sub-base of vehicle C, and only applies to vehicle C that has been modified by that user, without extending to other vehicles of the same model. When the user issues the command "turn on the colorful light" in vehicle C, the command "turn on the colorful light" is determined by matching the component name in the personalized knowledge sub-base, thus controlling the vehicle to turn on the ambient light.

[0140] Based on the above knowledge base, this application provides a control method 800. This control method uses the above knowledge base to help understand the user's intent, enabling the user to control the component to execute corresponding instructions through voice and multimodal reference, thereby improving the convenience and accuracy of the user's control of the component and helping to enhance the user experience.

[0141] Figure 8 shows a schematic flowchart of a control method 800 according to an embodiment of this application. As shown in Figure 8, the method 800 includes:

[0142] S810: Obtain first input from the user, which includes a description of the object the user is pointing to.

[0143] The first input indicates the user's intent. The first input can be voice input or text input, and can also be combined with gestures, eye direction, head posture, and other actions.

[0144] Optionally, when the first input includes user information and first location information, this step further includes: determining the first area where the user is located within the vehicle's cabin based on the user information. The method for determining the first area where the user is located can be found in method 300 above, and will not be repeated here.

[0145] S820: Determine keywords based on the user's first input.

[0146] Named entity recognition (NER) or other term analysis methods can be used to break down user statements into multiple keywords. Keyword splitting methods can include splitting based on linguistic information, rewriting using a large model, or using model slotting, etc. These various splitting methods can be freely combined according to actual needs.

[0147] Language information includes parts of speech and syntactic components. Parts of speech can include verbs, adjectives, nouns, adverbs, pronouns, numerals, etc. Common syntactic components include subjects, predicates, objects, attributives, subject-predicate phrases, etc. Pronouns can include words such as you, I, he, here, there, etc., that replace nouns, verbs, adjectives, or numerals.

[0148] The following examples from different scenarios illustrate how to determine keywords based on parts of speech or linguistic information:

[0149] Scenario 1: When a user wants to control a component, the common language structure is a combination of verb, modifier, and noun. Modifiers include adjectives and subject-verb phrases. For example, when a user wants to pause the rear door, they might say "Pause the rear door," which has a verb + modifier + noun structure; or, they might say "(Pause) the rear door," which has a (causative verb) + modifier + noun + verb structure; or, they might say "Pause the door behind me," which has a noun + verb + modifier structure. Regardless of the method, the user's command can be broken down into multiple keywords, including: verb - "pause," modifier - "behind me," and noun - "door."

[0150] Scenario 2: When users want answers to their vehicle's questions, the common language structure is a combination of attributive, noun, and interrogative adverb. For example, when a user doesn't know the name of a certain light inside the car, they might say, "What is the light in the car that can change color?" The language structure of this sentence is attributive + noun + interrogative adverb. This instruction can be broken down into several keywords, including: attributive - "in the car", attributive - "can change color", noun - "light", and interrogative adverb - "what is it".

[0151] Scenario 3: When users issue commands using voice and multimodal reference, they often substitute locative words with gestures, eye contact, and head posture. In this case, the common language structure is a combination of verbs, pronouns, modifiers, and nouns. For example, if a user wants to turn off the hazard lights, they might say "Turn off the red button" while pointing to the center hazard light button. The language structure of this sentence is verb + pronoun + modifier + noun. Alternatively, the user might say "Turn off the red one that's flashing" while pointing to the center hazard light button. The language structure of this sentence is verb + modifier + pronoun. Regardless of the method, the user's command can be broken down into multiple keywords, including: verb - "turn off", modifier - "red" or "flashing", and noun - "button" or "this one".

[0152] For expressions that deviate from common language norms, the user's non-standard statements can be rewritten into standard statements using a large language model. Then, wildcards can be used to split keywords according to the standard format. A wildcard is a word that can be fuzzily replaced by one or more characters. For example, wildcard A can be spatial location information, with enumerated values ​​including "above," "below," "in front," "left," etc.; wildcard B can be color, with enumerated values ​​including "red," "black," "white," etc.; and wildcard C can be action, with enumerated values ​​including "open," "close," "pause," etc. Alternatively, after rewriting the user's non-standard statements into standard statements, keywords can be further split based on language information.

[0153] In one implementation, key information from user intent can be extracted using slot entity extraction methods in a large language model. For example, by training a large model, slot information can be obtained, and slot filling can be performed using named entity recognition and slot prediction. For instance, if the first input is "pause the car door behind me," the large model recognizes the user's intent as "pause," further determines that the slots corresponding to this intent include "slot direction" and "slot device," and fills "slot direction" with "behind me" and "slot device" with "car door," thereby achieving keyword splitting.

[0154] S830: Classify keywords based on tag information in the knowledge base.

[0155] For example, keyword classification mainly targets the modifiers (including adjectives, subject-predicate phrases, etc.) or nouns in the keywords, while verbs and interrogative adverbs may not be classified.

[0156] Table 1 shows an example table of vehicle control keywords according to an embodiment of this application. As shown in Table 1, the keywords (e.g., modifiers and nouns) obtained in S820 can be further classified according to the tag information in the knowledge base, such as spatial location information (including vehicle body location words, relative component location words and relative user location words), name information (including component name, component name superordinate words, component name synonyms), function information, color information, shape information, material information, etc.

[0157] Table 1 Examples of Vehicle Control Keywords

[0158] For example, taking scenario one in S820 as an example, these extracted keywords can be further divided into: keywords representing spatial location information "behind me", more specifically, "behind me" is a relative user location term; and keywords representing component names "car door".

[0159] For example, taking scenario two in S820 as an example, these extracted keywords can be further divided into: the keyword "inside the car" which represents spatial location information, more specifically, "inside the car" is a vehicle body orientation word; the keyword "light" which represents component name, more specifically, "light" is a superordinate word of component name; and the keywords representing functional information include "can change color".

[0160] For example, taking scenario three in S820 as an example, these extracted keywords can be further divided into: keywords representing component names include "button" and "this", more specifically, "button" is a superordinate word of component name; keywords representing color information include "red"; keywords representing function information include "flashing".

[0161] S840: Match the categorized keywords with the corresponding tag information in the knowledge base one by one to determine the components.

[0162] As shown in Table 1, these keywords can be categorized into tags such as spatial location information, name information, functional information, color information, shape information, and material information. These keywords can be matched in the knowledge base according to a certain order or weight. For example, the matching degree between keywords and corresponding tag information in the knowledge base can be calculated using similarity. When a keyword matches all tag information, it is determined that the first input is a complete match with the tag information.

[0163] In one implementation, when the voice input is "Open the sunshade above my head," step S810 determines the user's location as the driver's seat, and step S820 determines the keywords to include the verb "open," the modifier "the one above my head (located at the top of the driver's seat)," and the noun "this sunshade." In step S830, based on the spatial location information in the knowledge base, the modifier "the one above my head (located at the top of the driver's seat)" is further classified as a relative user location locator in the spatial location information; based on the component name synonyms in the knowledge base, "this sunshade" is further classified as a component name synonym.

[0164] For example, the categorized keywords can be matched one by one with the corresponding tag information in the knowledge base according to a preset order. Taking the matching order of spatial location information-shape information-name information-function information-color information in the tag information as an example, combined with the spatial location information in the knowledge base, the location of the component to be controlled is determined by "the one above my head (the top of the driver's seat)," narrowing the matching range to components near the roof of the car; if the user does not describe the shape information, then the name information is matched further; in the knowledge base, the synonyms of the component name that match "this sunshade" are "sunshade curtain" or "sunshade panel," where the matching degree of "sunshade curtain" with "this sunshade" is higher than that of "sunshade panel," then the function information is matched further; if the user does not describe the function information, then the color information is matched further; if the user does not describe the color of the component, then the knowledge base matching ends, and the components that completely match the first input are determined to be "sunshade curtain" and "sunshade panel." Among them, the component with the highest matching degree with the first input is "sunshade curtain," and the component with the second highest matching degree is "sunshade panel."

[0165] For example, the categorized keywords and their corresponding tags in the knowledge base can also be weighted and sorted. Assuming the weights of each tag from largest to smallest are: spatial location information > shape information > name information > function information > color information, taking "Open the sunshade in front of my head (the one on the front side of the driver's seat)" as an example, based on the name information, the components matching "this sunshade" are "sunshade curtain" and "sunshade panel," with "sunshade curtain" matching "this sunshade" more closely than "sunshade panel." Based on the spatial location information, "sunshade panel" matches "this sunshade" more closely than "sunshade curtain." Therefore, the components that perfectly match the first input are "sunshade curtain" and "sunshade panel," and according to the weighted sorting calculation method, the component with the highest matching degree is determined to be "sunshade panel," and the component with the second highest matching degree is determined to be "sunshade curtain."

[0166] For example, in one possible scenario, if it is determined from the spatial location information that the sunshade of the vehicle is not located at the top front of the driver's seat, then the only component that is completely matched with the first input is the "sunshade panel", and the component with the second highest matching degree is the "sunshade".

[0167] In one possible scenario, if the first input is "Open the sunshade in front of me," according to the matching method described above, the spatial location information corresponding to "in front of me" is "in front of the driver's seat." The component names matching "this sunshade" are "sunshade panel" and "sunshade curtain," with "sunshade curtain" having a higher matching degree than "sunshade panel." However, according to the spatial location information, neither "sunshade panel" nor "sunshade curtain" is located in front of the driver's seat. Therefore, it is determined that the first input and the knowledge base do not match any component. Based on similarity, the component with the highest matching degree is "sunshade curtain," and the component with the second highest matching degree is "sunshade panel."

[0168] S850: Determine whether the component that perfectly matches the first input is unique.

[0169] For example, if there is only one component that completely matches the first input, step S860 is executed; otherwise, step S870 is executed.

[0170] S860: The only component that perfectly matches the first input is identified as the first component.

[0171] S870: Determine whether the number of components that perfectly match the first input is greater than 1.

[0172] For example, when the number of components that completely match the first input is greater than 1, step S871 is executed; otherwise, according to the above judgment steps, if the number of components that completely match the first input does not meet the conditions of being greater than 1 or equal to 1, that is, if no component that completely matches the first input is found, step S872 is executed.

[0173] S871: The control vehicle clarifies the user's choice and determines the first component based on the user's selection.

[0174] S872: Control the vehicle to make recommendations or inquiries to the user, and determine the first component based on the user's selection.

[0175] S880: Controls the vehicle to perform operations related to the first component.

[0176] When the number of components that perfectly match the first input is greater than one, clarification can be provided to the user based on the top two or three components with the highest matching degree. For example, when the top two matching components are "sunshade" and "glare shield," the display device can show the text "Do you want to open the sunshade or the glare shield?" or the speaker can announce "Do you want to open the sunshade or the glare shield?" The user can then input the corresponding text on the display device or answer "glare shield" via voice. In response to the user's selection, the vehicle opens the glare shield. Optionally, the display device can also show the user text such as "Okay, opening the glare shield for you," or similar text or images, and / or announce "Okay, opening the glare shield for you" via voice, informing the user of the specific operation performed by the vehicle.

[0177] In one implementation, if the user does not understand the difference between a sunshade and a sun visor, they may not directly answer the vehicle's clarification questions. For example, when asked, "Do you want to open the sunshade or the sun visor?", the user might ask, "Where is the sun visor?" or they might say, "Open this," and use gestures, eye contact, head posture, or other methods to indicate the location of the sun visor. The vehicle cannot directly determine the first component based on the user's feedback. In this case, the control flow of steps S810 to S880 can be re-executed for multiple rounds of clarification until the first component is finally determined.

[0178] It should be understood that the order of the vehicle opening the sunshade and the text display and voice broadcast in the above examples of the embodiments of this application is not limited, and these operations can be performed sequentially or simultaneously.

[0179] When no component is matched based on the first input and the knowledge base, the user can be asked or recommended based on the component with the highest matching score. For example, the first input is "Open the sunshade in front of me," and the spatial location information corresponding to "in front of me" is "in front of the driver's seat." The component names matching "this sunshade" are "sunshade panel" and "sunshade curtain," with "sunshade curtain" having a higher matching score than "sunshade panel." According to the spatial location information, neither "sunshade panel" nor "sunshade curtain" is located in front of the driver's seat; therefore, the number of components that perfectly match the first input is less than one, and the component with the highest matching score is "sunshade curtain." In one implementation, a recommendation can be made to the user, for example, by displaying the text "You can try opening the sunshade curtain" on a display device or by using a speaker to announce "You can try opening the sunshade curtain." The first component can be determined based on the text "sunshade curtain" entered by the user on the display device or by the voice "sunshade curtain." Optionally, the user can also be asked. For example, the first component can be determined based on the text "Do you want to open the sunshade?" displayed on the display device or the voice announcement "Do you want to open the sunshade?" made by the speaker.

[0180] It should be understood that the above example of asking users questions based on the first or second ranked component in terms of matching degree is only an example. The specific questioning method can be flexibly adjusted according to the actual situation. This application embodiment does not limit the selection range and method of components when asking questions.

[0181] The above control method 800 determines the first component by splitting the first input into keywords and matching these keywords with the tag information in the knowledge base, thereby enabling the user to accurately control the vehicle.

[0182] Optionally, the above control method can also be used to learn user habits, obtain user information based on user habits, and then use this user information to build a personalized knowledge sub-base. For example, when the user of vehicle D's first input is "turn on the colored lights," according to the above control method, it is determined that there is no component that completely matches the first input. The component with the highest matching degree is "ambient light," and the component with the second highest matching degree is "projection headlights." At this time, a voice announcement can be made using a speaker: "No colored lights found. Do you mean turn on the ambient lights or the projection headlights?" The user answers "ambient lights" in voice. Based on the user's selection, it can be learned that the user's usage habit is "colored lights = ambient lights." The component name "colored lights" is then stored in vehicle D's personalized knowledge sub-base, only applicable to vehicle D corresponding to this user, and not extended to other vehicles of the same model. When the user issues the command "turn on the colored lights" again in vehicle D, by matching the component names in the personalized knowledge sub-base, it is determined that "turn on the colored lights" is the same as the "turn on the ambient lights" command, and vehicle D is controlled to turn on the ambient lights.

[0183] In one implementation, the information in the knowledge base can also be combined with a large language model. For example, the information in the knowledge base can be used to fine-tune an existing large model, updating the model's parameters so that the instructions output by the large model can better match the user's intent.

[0184] For example, a knowledge base can be combined with retrieval-augmented generation (RAG) technology. For instance, information related to the user's intent can be retrieved from the knowledge base first, and this retrieved information can be used as contextual input or prompts for a large language model, which can then be combined to generate the final content.

[0185] Combining information from the knowledge base and a fine-tuned large model, embodiments of this application provide a control method 900, which includes:

[0186] S910: Obtain first input from the user, which includes a description of the object the user is pointing to.

[0187] Similar to steps S810 and S310 mentioned above, they will not be described again here.

[0188] S920: Extract text features based on the first input.

[0189] In one implementation, a text processor can be used to parse the first input to obtain text features, which can be features represented by numbers. For example, the text processor decomposes the text in the first input into multiple text units (or tokens), which are common character sequences, words, etc. in text; therefore, text units can be words, characters, phrases, numbers, etc. Further, the text processor represents each of the multiple text units using a text unit index, which can be in numeric form, so that the neural network can process the text unit index. Exemplarily, the text processor can be a natural language processing tool such as a text tokenizer.

[0190] For example, if the first input is "close the rear door", the text features may include: text cell indexes related to the action (such as text cell indexes related to "close"), text cell indexes related to spatial location information (such as text cell indexes related to "rear"), and text cell indexes related to the part name (such as text cell indexes related to "door").

[0191] For example, if the first input is "What is the black button with a red triangle on top?", then the text features may include: text unit indexes related to spatial location information (such as text unit indexes related to "on top"), text unit indexes related to shape information (such as text unit indexes related to "red triangle"), text unit indexes related to color information (such as text unit indexes related to "black"), text unit indexes related to part name (such as text unit indexes related to "button"), and text unit indexes related to interrogative adverbs (such as text unit indexes related to "what is it").

[0192] S930: Input text features into a large model for retrieval and rewriting to generate meaningful sentences.

[0193] For example, chunks similar to text features can be retrieved from a large model's database. This retrieval can be achieved by calculating similarity scores (e.g., cosine similarity). The retrieved results are then sorted based on their similarity scores, and the most similar chunk is selected to rewrite the first input, outputting the rewritten original sentence.

[0194] For example, if the first input is "close the rear door", based on the text unit index related to spatial location information and the text unit index related to component name, the fragment most similar to these text features is "tailgate". Combining the text unit index related to action, the first input is rewritten as "close the tailgate", that is, the intended statement is "close the tailgate".

[0195] For example, if the first input is "What is the black button with a red triangle on it?", based on the text unit index related to spatial location information, the text unit index related to shape information, and the text unit index related to color information, the fragment most similar to these text features is "danger warning light". Combining the text unit index related to interrogative adverbs, the first input is rewritten as "What is the danger warning light button?", that is, the original sentence is "What is the danger warning light button?".

[0196] S940: Determine the first component based on the intended meaning of the statement.

[0197] S950: Controls the vehicle to perform operations related to the first component.

[0198] Referring to Figure 2, the control methods 800 and 900 described above can be executed either in the cloud or on the vehicle. For example, when the network signal is poor at the vehicle's location, data transmission between the vehicle and the cloud may be delayed. In this scenario, control methods 800 and 900 can be executed on the vehicle first. Optionally, control method 900 requires higher computing power. Since different vehicle systems have different computing power, control method 900 can be executed on the cloud first, while control method 800 can be executed on the vehicle first.

[0199] In one implementation, when the user is asked a second question using the control method described above, after receiving an affirmative answer, the user's intent can be extracted as supplementary information for the knowledge base, thereby enabling real-time updates to the knowledge base.

[0200] Figure 9 shows a schematic diagram of an online knowledge base learning scheme according to an embodiment of this application. According to the control method 800 described above, during user-vehicle interaction, when the user's first input matches multiple components in the knowledge base or when no matching component exists in the knowledge base, the user is clarified, questioned, or recommended, thereby obtaining the user's true intent. Based on the user's true intent, the vehicle is controlled to perform corresponding operations. As shown in Figure 9, a data pair can be generated based on the user's description of the first component and the question result in the first input, and this data pair is saved as a new entry in the personalized knowledge sub-base. For example, this data pair can be <user statement, question result>. When a new entry is added to the personalized knowledge sub-base for the first time, knowledge upload is triggered, and the <user statement, question result> data pair is uploaded to the main knowledge base. If another vehicle's user also has the same description and corresponding component, then <user statement, question result> is also saved as a new entry in that user's personalized knowledge sub-base and uploaded to the main knowledge base. In this case, the main knowledge base records the <user statement, question result> data pair as having been uploaded twice. In one implementation, when the number of uploads of the <user statement, query result> data pair recorded in the knowledge base exceeds a preset number, a knowledge update can be performed. The <user statement, query result> data pair is then saved as supplementary data in the knowledge base and applied to all vehicles of that brand and model. After the knowledge base is updated, when the first input includes the above user statement, the query result corresponding to that user statement can be directly determined without further querying, thereby improving interaction efficiency and enhancing user experience.

[0201] For example, the knowledge base can be stored in the cloud and / or on the vehicle.

[0202] Based on the online learning scheme for the knowledge base in Figure 9, this application embodiment provides a knowledge base update method 1000. Figure 10 shows a schematic flowchart of a knowledge base update method 1000 according to an embodiment of this application, which can be executed by a cloud server. As shown in Figure 10, the method 1000 includes:

[0203] S1010: Establish the first association relationship between the first semantic and the first component in the first input in the personalized knowledge sub-base.

[0204] For example, after obtaining the user's first input "turn on the lane display lights" for vehicle E, and after multiple rounds of questioning and prompting, it is determined that the first semantic "lane display lights" in the first input is "projection headlights" in the knowledge base. Then, it is determined that in the user's usage habits, "lane display lights" and "projection headlights" are considered to be the same function. The first association <lane display lights, projection headlights> is obtained, and this first association is stored as supplementary information for the component name in the personalized knowledge sub-base of vehicle E, and the personalized knowledge sub-base is updated.

[0205] The first association above, <lane display lights, projector headlights>, only applies to the personalized knowledge sub-base of vehicle E that has been modified by this user, and will not be extended to other vehicles of the same model. When the user's second input for vehicle E is "turn off lane display lights", it can be determined that the component that perfectly matches "lane display lights" is "projector headlights", and vehicle E can be controlled to directly turn off the projector headlights.

[0206] S1020: Upload the first association to the knowledge base and record the number of times the first association has been uploaded.

[0207] For example, the knowledge base could be a knowledge base for all users, or a knowledge base for multiple regions. To facilitate the analysis of usage habits of multiple users, a knowledge upload can be triggered, uploading the first association from the personalized knowledge sub-base of vehicle E's user as a new entry to the knowledge base. The knowledge base can record these events. When vehicle E's personalized knowledge sub-base uploads the first association, the knowledge base records the upload count of that first association as 1.

[0208] For example, when the second input of the user of vehicle E also includes the first semantic "display lane lights", the first association already exists in the user's personalized knowledge sub-base, so there is no need to upload the first association to the main knowledge base.

[0209] S1030: A first association is established in the knowledge base when any of the following conditions are met: the number of uploads of the first association is greater than or equal to a first threshold, the ratio between the number of uploads and the total number of vehicles is greater than or equal to a second threshold, and the ratio between the number of uploads and the total number of times the user describes the first component through the fourth input is greater than or equal to a third threshold.

[0210] In one implementation, the knowledge base can be enriched by acquiring voice or text input related to the first component from multiple users, and this enrichment can be achieved through data feedback, thus enabling real-time updates to the knowledge base. The fourth input includes the first semantic meaning, but it can also be the first input or the second input.

[0211] For example, besides the user of vehicle E, data pairs generated by other users of the same brand and model during interaction can also be uploaded to the knowledge base. After the first association is uploaded to the personalized knowledge sub-base of vehicle E, if it is determined that other users' personalized knowledge sub-bases have also uploaded the same first association, the upload count of the first association can be recorded. For example, if the personalized knowledge sub-base of vehicle F also uploads the first association, the knowledge base records the upload count of the first association as 2 times. The knowledge base can be updated by adding <lane display light, projection headlights> as supplementary information for the component name when any of the following conditions are met: the upload count of the first association is greater than or equal to a first threshold; the ratio of the upload count to the total number of vehicles is greater than or equal to a second threshold; and the ratio of the upload count to the total number of times the user describes the first component through the fourth input is greater than or equal to a third threshold. After the knowledge base is updated, the updated knowledge base can be sent to vehicles of the same brand and model. When any user gives the voice instruction "turn on the lane display light," the vehicle can be controlled to directly turn on the projection headlights.

[0212] In one implementation, a first association in the personalized knowledge sub-base can be combined with the user's profile information. The profile information may include one or more of the user's gender, age, region, language, etc., and can be obtained through information entered by the user when purchasing a vehicle or user data collected by the car manufacturer. When uploading the first association to the personalized knowledge sub-base, the user's profile information can also be uploaded simultaneously, establishing a second association combining the profile information and the first association. In subsequent knowledge queries, the query can be performed based on the second association, which can be <first semantic, query result, profile information>.

[0213] It should be noted that if the information processed in this application involves users' personal information (such as facial images, voiceprints and other biometric information, location trajectory information and personal profile information, usage habits, etc.), the processing of users' personal information will be based on legality, and users will be fully informed and authorized, in accordance with the relevant laws and regulations on personal information protection in the country or region where the application is made.

[0214] After updating the regional knowledge base or the general knowledge base in the cloud, the regional knowledge base and the general knowledge base can be distributed to the corresponding vehicle terminals.

[0215] For example, after recognizing the instruction "Turn on the lane display light" from user H in region G, the second association <lane display light, projector headlight, region G> is obtained and uploaded to the knowledge base. When the number of uploads of the second association is greater than or equal to a first threshold, or the ratio of this upload count to the total number of vehicles is greater than or equal to a second threshold, or the ratio of the upload count to the total number of times all users describe the first component via the fourth input is greater than or equal to a third threshold, the second association <lane display light, projector headlight, region G> can be added to the knowledge base as supplementary information for the component name. When user I in region G issues the instruction "Turn off the lane display light," by obtaining user I's region information and the first semantic "lane display light" in the instruction, it can be determined that the user's profile information includes region G. Therefore, during knowledge queries, the user's region information, the first semantic "lane display light," and the second association in the knowledge base can be matched to determine that the "lane display light" mentioned by the user is "projector headlight."

[0216] Optionally, the overall knowledge base can be divided into multiple regional knowledge bases based on the user's geographic information, with each regional knowledge base applied to vehicles in its corresponding region. Taking the regional knowledge base for the first region as an example, a second association can be established in the regional knowledge base when any of the following conditions are met: the number of uploads of the second association to the regional knowledge base of the first region is greater than or equal to a fourth threshold; the ratio between the number of uploads and the total number of vehicles in the first region is greater than or equal to a fifth threshold; and the ratio between the number of uploads and the total number of times users in the first region control the first component through a third input is greater than or equal to a sixth threshold.

[0217] For example, a user H in region G issues the command "Turn on the lane display light," retrieves the second association <lane display light, projector headlight, region G>, and uploads it to the region G knowledge base. When the number of uploads of the second association to the region G knowledge base is greater than or equal to the fourth threshold, or the ratio between the number of uploads and the total number of vehicles in the region is greater than or equal to the fifth threshold, or the ratio between the number of uploads and the total number of times users in the region control the first component through the third input is greater than or equal to the sixth threshold, the second association <lane display light, projector headlight, region G> can be added to the region G knowledge base as supplementary information for the component name. When user I in region G issues the command "turn off the lane lights," by obtaining user I's region information and the first semantic meaning of "lane lights" in the command, it can be determined that the user's profile information includes region G. Therefore, during knowledge querying, the user's region information and the first semantic meaning of "lane lights" can be matched with the second association relationship in the region G knowledge base to determine that the "lane lights" mentioned by the user are "projection headlights." However, when user K in region J issues the command "turn off the lane lights," by obtaining user K's region information and the first semantic meaning of "lane lights" in the command, it can be determined that the user's profile information does not include region G. Therefore, it is impossible to match the user's "lane lights" with "projection headlights." Methods 800 or 900 mentioned above can be used to determine whether the "lane lights" mentioned by user K are "projection headlights."

[0218] For example, the regional knowledge base may also include the association between the first semantic and dialect information in the first region and the first component. This dialect information may be the user's accent or the user's specific description of the first component. The user's specific description of the first component can be updated in the regional knowledge base as the first semantic using the aforementioned method, and the user's accent can be recognized by the voice interaction system.

[0219] The user's region can be determined based on the dialect information. After the vehicle receives the regional knowledge base for the first region from the cloud server, when the user's third input includes the dialect information, it can be determined that the user's region is the first region. Then, based on the dialect information and the regional knowledge base corresponding to the first region, the first component can be determined, thereby controlling the vehicle to perform operations related to the first component.

[0220] It should be understood that the first threshold, second threshold, third threshold, fourth threshold, fifth threshold and sixth threshold mentioned above can be set or adjusted based on empirical data, and the embodiments of this application do not limit this.

[0221] Figure 11 shows a schematic block diagram of a control device 1100 provided in an embodiment of this application. The control device 1100 includes: an acquisition unit 1110, configured to acquire a first input from a user, the first input including a description of an object pointed to by the user; a determination unit 1120, configured to determine, based on the first input and a knowledge base, that the object pointed to by the user is a first component; and a control unit 1130, configured to control a vehicle to perform an operation related to the first component. The knowledge base includes spatial location information, which includes at least one of the following: the orientation information of one or more components relative to the vehicle, the orientation information of the first component relative to a second component, and the orientation information of the first component relative to the user. The one or more components include the first component, and the vehicle includes one or more components.

[0222] Optionally, the first input may include first orientation information. The determining unit 1120 is specifically used to: determine the first component based on the first orientation information and a knowledge base.

[0223] Optionally, the first input can be voice input, and the determining unit 1120 is specifically used to: obtain the text content corresponding to the voice input; and determine the first location information based on the text content.

[0224] Optionally, the knowledge base also includes at least one of the following: name information, function information, color information, shape information, and material information of the first component. The name information of the first component includes the name of the first component and one or more names that match the name of the first component. The determining unit 1120 is further configured to determine that the second information included in the first input matches at least one of the following: name information, function information, color information, shape information, and material information of the first component. The second information is a description by the user of at least one dimension of the name, function, color, shape, and material of the first component.

[0225] Optionally, the first input includes first orientation information and second information; the determining unit 1120 is specifically used to: determine the first component based on the first orientation information, the second information and the knowledge base.

[0226] Optionally, the first input includes second information; the determining unit 1120 is specifically used to: determine the first component based on the second information and the knowledge base.

[0227] Optionally, the first input includes user information and first orientation information; the determining unit 1120 is specifically used to: determine the first area where the user is located in the vehicle cabin based on the user information; and determine the first component based on the first area, the first orientation information, and the orientation information of the first component relative to the user.

[0228] Optionally, the first input includes information about the second component and first orientation information; the determining unit 1120 is specifically used to: determine the first component based on the information about the second component, the first orientation information, and the orientation information of the first component relative to the second component.

[0229] Optionally, the first input includes vehicle information and first orientation information; the determining unit 1120 is specifically used to: determine the first component based on the vehicle information, the first orientation information and the orientation information of the first component relative to the vehicle.

[0230] Optionally, the determining unit 1120 is specifically configured to: determine multiple components, including the first component, based on the first input and the knowledge base; and determine the first component from the multiple components based on at least one of the user's gaze direction, gesture input, and head posture when the first input is obtained.

[0231] Optionally, the determining unit 1120 is specifically used to: determine multiple components based on the first input and the knowledge base, the multiple components including the first component; the control unit 1130 is further used to control the vehicle to prompt the user to select at least one of the multiple components; the determining unit 1120 is specifically used to: determine the first component based on the user's selection result.

[0232] Optionally, before determining the first component based on the user's selection, the control unit 1130 is further configured to: based on a knowledge base, control the vehicle to prompt the user with difference information between multiple components, the difference information including differences in at least one dimension of the multiple components in terms of spatial location information, functional information, color information, shape information, and material information.

[0233] Optionally, the control unit 1130 is further configured to: when no component is matched according to the first input and the knowledge base, control the vehicle to prompt the user for at least one component; the determining unit 1120 is specifically configured to: determine the first component based on the user's selection result for at least one component.

[0234] In one embodiment, the control device 1100 may further include a setup unit.

[0235] Optionally, the knowledge base includes a first knowledge base, which is a knowledge base for the vehicle. The device further includes: an establishment unit, configured to establish a first association relationship between the first semantic and the first component in the first knowledge base; an acquisition unit 1110, further configured to acquire a second input from the user, the second input including the first semantic; and a determination unit 1120, further configured to determine the first component based on the second input and the first knowledge base.

[0236] In one embodiment, the control device 1100 may further include a receiving unit.

[0237] Optionally, the knowledge base includes a second knowledge base, and the device further includes: a receiving unit, configured to receive a second knowledge base for a first region sent by a cloud server, the second knowledge base including the association relationship between dialect information of the first semantic in the first region and the first component; an acquisition unit 1110, further configured to acquire a third input from the user, the third input including dialect information; and a determining unit 1120, further configured to determine the first component based on the dialect information and the second knowledge base.

[0238] For example, the acquisition unit 1110 can be the computing platform 120 in Figure 1, or the processing circuit, processor, or controller in the computing platform 120. Taking the processor 121 in the computing platform as an example, the acquisition unit 1110 can acquire the first input from the user.

[0239] For example, the determining unit 1120 can be the computing platform 120 in Figure 1, or the processing circuit, processor, or controller in the computing platform 120. Taking the determining unit 1120 as the processor 122 in the computing platform as an example, after the processor 121 obtains the first input from the user, the processor 122 can determine the object pointed to by the user as the first component based on the first input and the knowledge base.

[0240] For example, the control unit 1130 may be the computing platform 120 in Figure 1, or the processing circuit, processor, or controller in the computing platform 120. Taking the control unit 1130 as the processor 123 in the computing platform as an example, after the processor 122 determines the first component, the processor 123 can control the vehicle to perform operations related to the first component.

[0241] For example, the establishment unit can be the computing platform 120 in Figure 1, or the processing circuit, processor, or controller in the computing platform 120. Taking the processor 124 in the computing platform as an example, the processor 124 can establish a first association relationship between the first semantic and the first component in the first knowledge base based on the first input obtained by the processor 121.

[0242] For example, the receiving unit can be the computing platform 120 in Figure 1, or the processing circuit, processor, or controller in the computing platform 120. Taking the processor 125 in the computing platform as the establishing unit as an example, the processor 125 can receive the second knowledge base for the first region sent by the cloud server.

[0243] The functions implemented by the acquisition unit 1110, the control unit 1130, the determination unit 1120, the establishment unit, and the receiving unit can be implemented by different processors, or by the same processor, or some functions can be implemented by the same processor. This application embodiment does not limit this.

[0244] It should be understood that the division of units in the above device is only a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, the units in the device can be implemented by a processor calling software; for example, the device includes a processor connected to memory, which stores instructions. The processor calls the instructions stored in memory to implement any of the above methods or to implement the functions of each unit in the device. The processor can be, for example, a general-purpose processor, such as a CPU or microprocessor, and the memory can be internal or external to the device. Alternatively, the units in the device can be implemented as hardware circuits. The functions of some or all units can be implemented through the design of the hardware circuits, which can be understood as one or more processors. For example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all units are implemented through the design of the logical relationships between the components within the circuit. In another implementation, the hardware circuit can be implemented using a PLD, such as an FPGA, which can include a large number of logic gates. The connection relationships between the logic gates are configured through configuration files, thereby implementing the functions of some or all units. All units of the above devices can be implemented entirely through processor calling software, or entirely through hardware circuits, or partially through processor calling software with the remaining parts implemented through hardware circuits.

[0245] In this application embodiment, a processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction reading and execution capabilities, such as a CPU, microprocessor, GPU, or DSP. In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. These logical relationships are fixed or reconfigurable. For example, the processor may be a hardware circuit implemented as an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the processor loading instructions to implement the functions of some or all of the above units. Furthermore, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, or DPU.

[0246] As can be seen, each unit in the above device can be one or more processors (or processing circuits) configured to implement the above methods, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0247] Furthermore, the units in the above devices can be integrated in whole or in part, or they can be implemented independently. In one implementation, these units are integrated together as a System-on-a-Chip (SoC). The SoC may include at least one processor for implementing any of the above methods or implementing the functions of the units in the device. The at least one processor may be of different types, such as CPU and FPGA, CPU and AI processor, CPU and GPU, etc.

[0248] This application also provides a control device, which includes: a memory for storing a computer program; and a processor for executing the computer program stored in the memory, so that the device can execute any of the control methods described in the above embodiments.

[0249] Alternatively, if the device is located in a vehicle, the processor may be the processor 121-12n shown in FIG1.

[0250] This application also provides a vehicle that may include the control device 1100 described above.

[0251] This application also provides a computer-readable medium storing instructions that, when executed by a processor, cause the processor to implement any of the control methods described in the above embodiments.

[0252] This application also provides a computer program product, which includes computer program code. When the computer program code is run on a computer, it causes the computer to execute any of the control methods described in the above embodiments.

[0253] This application also provides a chip that includes a circuit that can be used to execute any of the control methods described in the above embodiments.

[0254] In implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software. The method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or by a combination of hardware and software modules within the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, power-on erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0255] It should be understood that in the embodiments of this application, the memory may include read-only memory and random access memory, and provides instructions and data to the processor.

[0256] It should also be understood that, in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0257] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0258] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0259] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0260] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0261] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0262] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0263] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A control method, characterized in that, The method includes: Obtain first input from the user, the first input including the user's description of the object they are pointing to; Based on the first input and the knowledge base, the object pointed to by the user is determined to be the first component; Controlling the vehicle to perform operations related to the first component, wherein the knowledge base includes spatial location information, which includes at least one of the following: the orientation information of one or more components relative to the vehicle, the orientation information of the first component relative to a second component, and the orientation information of the first component relative to the user; the one or more components include the first component, and the vehicle includes the one or more components.

2. The method according to claim 1, characterized in that, The first input includes first location information. The step of determining that the object pointed to by the user is the first component based on the first input and the knowledge base includes: The first component is determined based on the first location information and the knowledge base.

3. The method according to claim 2, characterized in that, The first input is voice input. Determining the first component based on the first location information and the knowledge base includes: Obtain the text content corresponding to the voice input; The first location information is determined based on the text content.

4. The method according to any one of claims 1 to 3, characterized in that, The knowledge base also includes at least one of the following: name information, function information, color information, shape information, and material information of the first component. The name information of the first component includes the name of the first component and one or more names that match the name of the first component. The method further includes: The second information included in the first input is determined to match at least one of the name information, function information, color information, shape information, and material information of the first component, wherein the second information is a description by the user of at least one dimension of the name, function, color, shape, and material of the first component.

5. The method according to claim 4, characterized in that, The first input includes first location information and second information; The step of determining that the object pointed to by the user is the first component based on the first input and the knowledge base includes: The first component is determined based on the first location information, the second information, and the knowledge base.

6. The method according to any one of claims 1 to 5, characterized in that, The first input includes user information and first location information; The step of determining that the object pointed to by the user is the first component based on the first input and the knowledge base includes: Based on the user's information, determine the first area where the user is located within the vehicle's cabin; The first component is determined based on the first region, the first orientation information, and the orientation information of the first component relative to the user.

7. The method according to any one of claims 1 to 5, characterized in that, The first input includes information about the second component and first orientation information; The step of determining that the object pointed to by the user is the first component based on the first input and the knowledge base includes: The first component is determined based on the information of the second component, the first orientation information, and the orientation information of the first component relative to the second component.

8. The method according to any one of claims 1 to 5, characterized in that, The first input includes vehicle information and first location information; The step of determining that the object pointed to by the user is the first component based on the first input and the knowledge base includes: The first component is determined based on the vehicle information, the first orientation information, and the orientation information of the first component relative to the vehicle.

9. The method according to any one of claims 1 to 8, characterized in that, The step of determining that the object pointed to by the user is the first component based on the first input and the knowledge base includes: Based on the first input and the knowledge base, multiple components are determined, including the first component; The first component is determined from the plurality of components based on at least one of the user's gaze direction, gesture input, and head posture when the first input is obtained.

10. The method according to any one of claims 1 to 8, characterized in that, The step of determining that the object pointed to by the user is the first component based on the first input and the knowledge base includes: Based on the first input and the knowledge base, multiple components are determined, including the first component; The vehicle is controlled to prompt the user to select at least one of the plurality of components; The first component is determined based on the user's selection.

11. The method according to claim 10, characterized in that, Before determining the first component based on the user's selection, the method further includes: Based on the knowledge base, the vehicle is controlled to prompt the user with difference information between the multiple components. The difference information includes differences in at least one dimension of the multiple components, namely spatial location information, functional information, color information, shape information, and material information.

12. The method according to any one of claims 1 to 8, characterized in that, The step of determining that the object pointed to by the user is the first component based on the first input and the knowledge base includes: When no component is matched based on the first input and the knowledge base, the vehicle is controlled to prompt the user with at least one component. The first component is determined based on the user's selection of the at least one component.

13. The method according to any one of claims 9 to 12, characterized in that, The knowledge base includes a first knowledge base, which is a knowledge base specific to the vehicle, and the method further includes: Establish a first association relationship between the first semantic in the first input and the first component in the first knowledge base; Obtain the user's second input, the second input including the first semantic; The first component is determined based on the second input and the first knowledge base.

14. The method according to any one of claims 9 to 12, characterized in that, The method further includes: Receive a second knowledge base for the first region sent by the cloud server. The second knowledge base includes the association relationship between the dialect information of the first semantic in the first region and the first component in the first input. Obtain the user's third input, which includes the dialect information; The first component is determined based on the dialect information and the second knowledge base.

15. A control device, characterized in that, The device includes: An acquisition unit is configured to acquire first input from a user, the first input including a description of the object the user is pointing to; The determining unit is configured to determine, based on the first input and the knowledge base, that the object pointed to by the user is the first component; A control unit is used to control the vehicle to perform operations related to the first component, wherein the knowledge base includes spatial location information, which includes at least one of the following: the orientation information of one or more components relative to the vehicle, the orientation information of the first component relative to a second component, and the orientation information of the first component relative to the user; the one or more components include the first component, and the vehicle includes the one or more components.

16. The apparatus according to claim 15, characterized in that, The first input includes first location information, and the determining unit is specifically used for: The first component is determined based on the first location information and the knowledge base.

17. The apparatus according to claim 16, characterized in that, The first input is voice input, and the determining unit is specifically used for: Obtain the text content corresponding to the voice input; The first location information is determined based on the text content.

18. The apparatus according to any one of claims 15 to 17, characterized in that, The knowledge base also includes at least one of the following: name information, function information, color information, shape information, and material information of the first component. The name information of the first component includes the name of the first component and one or more names that match the name of the first component. The determining unit is further configured to: The second information included in the first input is determined to match at least one of the name information, function information, color information, shape information, and material information of the first component, wherein the second information is a description by the user of at least one dimension of the name, function, color, shape, and material of the first component.

19. The apparatus according to claim 18, characterized in that, The first input includes first location information and second information; The determining unit is specifically used for: The first component is determined based on the first location information, the second information, and the knowledge base.

20. The apparatus according to any one of claims 15 to 19, characterized in that, The first input includes user information and first location information; The determining unit is specifically used for: Based on the user's information, determine the first area where the user is located within the vehicle's cabin; The first component is determined based on the first region, the first orientation information, and the orientation information of the first component relative to the user.

21. The apparatus according to any one of claims 15 to 19, characterized in that, The first input includes information about the second component and first orientation information; The determining unit is specifically used for: The first component is determined based on the information of the second component, the first orientation information, and the orientation information of the first component relative to the second component.

22. The apparatus according to any one of claims 15 to 19, characterized in that, The first input includes vehicle information and first location information; The determining unit is specifically used for: The first component is determined based on the vehicle information, the first orientation information, and the orientation information of the first component relative to the vehicle.

23. The apparatus according to any one of claims 15 to 22, characterized in that, The determining unit is specifically used for: Based on the first input and the knowledge base, multiple components are determined, including the first component; The first component is determined from the plurality of components based on at least one of the user's gaze direction, gesture input, and head posture when the first input is obtained.

24. The apparatus according to any one of claims 15 to 22, characterized in that, The determining unit is specifically used for: Based on the first input and the knowledge base, multiple components are determined, including the first component; The control unit is further configured to: control the vehicle to prompt the user to select at least one of the plurality of components; The determining unit is specifically used to: determine the first component based on the user's selection result.

25. The apparatus according to claim 24, characterized in that, Before determining the first component based on the user's selection, the control unit is further configured to: Based on the knowledge base, the vehicle is controlled to prompt the user with difference information between the multiple components. The difference information includes differences in at least one dimension of the multiple components, namely spatial location information, functional information, color information, shape information, and material information.

26. The apparatus according to any one of claims 15 to 22, characterized in that, The control unit is also used for: When no component is matched based on the first input and the knowledge base, the vehicle is controlled to prompt the user with at least one component. The determining unit is specifically used to: determine the first component based on the user's selection result for the at least one component.

27. The apparatus according to any one of claims 23 to 26, characterized in that, The knowledge base includes a first knowledge base, which is a knowledge base specific to the vehicle, and the device further includes: A unit is established to establish a first association relationship between the first semantic in the first input and the first component in the first knowledge base; The acquisition unit is further configured to acquire the user's second input, the second input including the first semantic; The determining unit is further configured to determine the first component based on the second input and the first knowledge base.

28. The apparatus according to any one of claims 23 to 26, characterized in that, The device further includes: The receiving unit is used to receive a second knowledge base for a first region sent by a cloud server. The second knowledge base includes the association relationship between the dialect information of the first semantic in the first region and the first component in the first input. The acquisition unit is further configured to acquire the user's third input, the third input including the dialect information; The determining unit is further configured to determine the first component based on the dialect information and the second knowledge base.

29. A control device, characterized in that, The device includes: Memory, used to store computer programs; A processor for executing a computer program stored in the memory to cause the apparatus to perform the method as described in any one of claims 1 to 14.

30. A vehicle, characterized in that, Includes the apparatus as described in any one of claims 15 to 29.

31. A computer-readable storage medium, characterized in that, It stores instructions that, when executed by a processor, cause the processor to implement the method as described in any one of claims 1 to 14.

32. A computer program product, characterized in that, The computer program product includes computer program code that, when run on a computer, causes the computer to perform the method as described in any one of claims 1 to 14.

33. A chip, characterized in that, The chip includes circuitry for performing the method as described in any one of claims 1 to 14.

Images