Electronic device mounted on vehicle and vehicle control method using same

Abstract

A disclosed embodiment relates to an electronic device mounted on a vehicle. The electronic device may: acquire a plurality of vehicle interior images by means of a plurality of cameras; acquire, from each of the plurality of vehicle interior images by using a plurality of neural network models, a biometric information set corresponding to each of the plurality of vehicle interior images and indicating biometric information of a driver of the vehicle; acquire, by using a prompt generation model, a prompt set including at least one prompt indicating, with text, a biometric state of the driver by combining one or more pieces of the biometric information included in the biometric information set; acquire, by using a generative artificial intelligence model, a driving state of the driver of the vehicle corresponding to the prompt set; on the basis of the driving state of the driver corresponding to each of the plurality of vehicle interior images, identify whether the driver of the vehicle is driving normally; and issue a warning on the basis that it is identified that the driver of the vehicle is not driving normally.

Description

Electronic device mounted on a vehicle and a method of controlling the vehicle through the same

[0001] The present disclosure relates to an electronic device mounted on a vehicle and a method for controlling the vehicle through the electronic device mounted on the vehicle. More specifically, it relates to a method for determining the state of a driver through an electronic device mounted on a vehicle and an electronic device for the same.

[0002] A driver monitoring system (DMS) is a system that uses a camera to capture images of the driver and recognizes their face to perform actions such as detecting drowsy driving or issuing driver status warnings. It is also utilized in intelligent speed assist systems, reverse assist systems, event data recorders (EDRs), or emergency braking systems. Recently, legislation is being enacted worldwide to mandate the installation of driver monitoring systems in vehicles for the purpose of driver protection and the prevention of road traffic accidents.

[0003] Meanwhile, driver monitoring systems that use a single camera to film the driver have limitations in accurately recognizing the driver's condition and require improvement.

[0004] According to one embodiment of the present disclosure, an electronic device mounted on a vehicle may be provided. According to one embodiment, the electronic device may include a plurality of cameras. According to one embodiment, the electronic device may include a memory for storing at least one instruction. According to one embodiment, the electronic device may include at least one processor including a circuit device. By executing at least one instruction individually or collectively by the at least one processor, the electronic device may acquire a plurality of in-vehicle images through the plurality of cameras. By executing at least one instruction individually or collectively by the at least one processor, the electronic device may acquire a set of biometric information corresponding to each of the plurality of in-vehicle images and representing biometric information of the driver of the vehicle from each of the plurality of in-vehicle images using a plurality of neural network models. By executing at least one instruction individually or collectively by the at least one processor, the electronic device can obtain a set of prompts including at least one prompt that represents the driver's biometric state as text by combining at least one biometric information included in the set of biometric information using a prompt generation model. By executing at least one instruction individually or collectively by the at least one processor, the electronic device can obtain the driving state of the driver of the vehicle corresponding to the set of prompts using a generative artificial intelligence model.By executing at least one instruction individually or collectively by the at least one processor, the electronic device can identify whether the driver of the vehicle is driving normally based on the driving state of the driver corresponding to each of the plurality of vehicle interior images. By executing at least one instruction individually or collectively by the at least one processor, the electronic device can output a warning based on the identification that the driver of the vehicle is not driving normally.

[0005] According to one aspect of the present disclosure, a method for controlling a vehicle through an electronic device mounted on the vehicle may be provided. The method according to one embodiment may include the operation of acquiring a plurality of in-vehicle images through a plurality of cameras. The method may include the operation of acquiring a set of biometric information corresponding to each of the plurality of in-vehicle images and representing biometric information of the driver of the vehicle from each of the plurality of in-vehicle images using a plurality of neural network models. The method may include the operation of acquiring a set of prompts including at least one prompt representing the driver's biometric state as text by combining at least one biometric information included in the set of biometric information using a prompt generation model. The method may include the operation of acquiring the driving state of the driver of the vehicle corresponding to the set of prompts using a generative artificial intelligence model. The method may include the operation of identifying whether the driver of the vehicle is driving normally based on the driving state of the driver corresponding to each of the plurality of in-vehicle images. The above method may include an operation of outputting a warning based on the identification that the driver of the vehicle is not driving normally.

[0006] According to one aspect of the present disclosure, a computer-readable recording medium may be provided that records a program for executing any one of the methods of controlling a vehicle through an electronic device mounted on a vehicle described above and below.

[0007] The present invention can be easily understood from the combination of the following detailed description and the accompanying drawings, where reference numerals denote structural elements.

[0008] Figure 1 is a conceptual diagram illustrating the components of an electronic device mounted in a vehicle and the operation of the components.

[0009] FIG. 2 is a block diagram illustrating the configuration of an electronic device mounted in a vehicle according to one embodiment of the present disclosure.

[0010] FIG. 3 is a flowchart illustrating a method for controlling a vehicle through an electronic device according to one embodiment of the present disclosure.

[0011] FIG. 4 is a reference diagram for explaining a method in which an electronic device acquires multiple vehicle interior images through a plurality of cameras according to one embodiment of the present disclosure.

[0012] FIG. 5a is a flowchart illustrating a method for an electronic device included in operation 320 according to one embodiment of the present disclosure to acquire a set of bio-information using a plurality of neural network models.

[0013] FIG. 5b is a reference diagram illustrating a method for an electronic device to acquire a first set of bio-information based on a first internal image using a plurality of neural network models according to one embodiment of the present disclosure.

[0014] FIG. 5c is a reference diagram showing a first set of bio-information according to one embodiment of the present disclosure.

[0015] FIG. 6a is a flowchart illustrating a method for an electronic device included in operation 330 according to one embodiment of the present disclosure to obtain a prompt using a prompt generation model.

[0016] FIG. 6b is a reference diagram for illustrating a prompt set according to one embodiment of the present disclosure.

[0017] FIG. 7a is a flowchart for explaining in detail a method for an electronic device according to one embodiment of the present disclosure to acquire the driving state of a vehicle driver corresponding to each of a plurality of vehicle interior images.

[0018] FIG. 7b is a flowchart for explaining in detail a method for an electronic device according to one embodiment of the present disclosure to acquire the driving state of a vehicle driver corresponding to each of a plurality of vehicle interior images.

[0019] FIG. 8a is a diagram for explaining in detail the operation of an electronic device according to one embodiment of the present disclosure identifying whether a driver of a vehicle is driving normally based on the driving state of the driver corresponding to each of a plurality of vehicle interior images.

[0020] FIG. 8b is a diagram for explaining in detail the operation of an electronic device according to one embodiment of the present disclosure identifying whether a driver of a vehicle is driving normally based on the driving state of the driver corresponding to each of a plurality of vehicle interior images.

[0021] FIG. 9 is a flowchart illustrating the operation of an electronic device according to one embodiment of the present disclosure to determine whether the heart rate of a vehicle driver falls within a normal range.

[0022] FIG. 10 is a flowchart illustrating the operation of an electronic device according to one embodiment of the present disclosure to output a warning based on the identification that the driver of a vehicle is not in normal driving.

[0023] FIGS. 11a to 11d are reference drawings for explaining the operation of an electronic device outputting a warning according to one embodiment of the present disclosure.

[0024] FIG. 12 is a block diagram illustrating the operation of controlling devices included in a vehicle as an electronic device according to one embodiment of the present disclosure identifies that the driver is not driving normally.

[0025] FIG. 13 is a diagram illustrating an operation performed using artificial intelligence technology in a disclosed embodiment.

[0026] FIG. 14 is a drawing illustrating a disclosed embodiment in which the electronic device of the present disclosure operates in conjunction with a server.

[0027] FIG. 15 is a drawing for explaining FIG. 14 in detail.

[0028] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments.

[0029] In relation to the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of the noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise.

[0030] In this document, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "at least one of A, B, or C" may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof.

[0031] The term “and / or” includes a combination of multiple related described components or any of the multiple related described components.

[0032] Terms such as "first," "second," or "first" or "second" may be used simply to distinguish a component from another component and do not limit the components in other aspects (e.g., importance or order).

[0033] Where any (e.g., 1st) component is referred to as "coupled" or "connected" to another (e.g., 2nd) component, with or without the terms "functionally" or "communicationly," it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0034] Terms such as “include” or “have” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in this document, and do not preclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0035] When it is said that one component is “connected,” “combined,” “supported,” or “in contact” with another component, this includes not only cases where the components are directly connected, combined, supported, or in contact, but also cases where they are indirectly connected, combined, supported, or in contact through a third component.

[0036] When it is said that a component is located “on” another component, this includes not only cases where one component is in contact with the other, but also cases where another component exists between the two components.

[0037] It should be understood that the blocks in each flowchart and combinations of flowcharts can be executed by one or more computer programs containing computer-executable instructions. One or more computer programs may be stored all in a single memory or may be partitioned and stored in multiple different memories.

[0038] One embodiment of the present disclosure may be represented by functional block configurations and various processing steps. Some or all of these functional blocks may be implemented by various numbers of hardware and / or software configurations that execute specific functions. For example, the functional blocks of the present disclosure may be implemented by one or more microprocessors or by circuit configurations for a specific function. Additionally, for example, the functional blocks of the present disclosure may be implemented in various programming or scripting languages. The functional blocks may be implemented as algorithms executed on one or more processors. Furthermore, the present disclosure may employ prior art for electronic configuration, signal processing, and / or data processing, etc.

[0039] All functions or operations, including functions related to artificial intelligence according to the present disclosure, are operated through a processor and memory. The processor may be composed of one or more processors. A single processor or a combination of processors may include circuitry that performs processing, such as an AP (Application Processor), CP (Communication Processor), GPU (Graphical Processing Unit), NPU (Neural Processing Unit), MPU (Microprocessor Unit), SoC (System on Chip), IC (Integrated Chip), etc.

[0040] In the present disclosure, a "vehicle" is a means of transportation that travels on a road or a track. A vehicle may be a concept that includes an internal combustion engine vehicle equipped with an engine as a power source, a hybrid vehicle equipped with an engine and an electric motor as a power source, an electric vehicle equipped with an electric motor as a power source, and the like. In one embodiment of the present disclosure, a vehicle may include at least one of an automobile, a train, and a motorcycle.

[0041] The present disclosure will be described in detail below with reference to the attached drawings.

[0042] Figure 1 is a conceptual diagram illustrating the components of an electronic device mounted in a vehicle and the operation of the components.

[0043] Referring to FIG. 1, an electronic device (100) may be placed inside a vehicle. The electronic device (100) may include a camera (110). The camera (110) may include a plurality of cameras (110-1 to 110-3). Although not shown in FIG. 1, the electronic device (100) may include additional components in addition to the plurality of cameras (110-1 to 110-3). The number and placement locations of the plurality of cameras (110-1 to 110-3) shown in FIG. 1 are exemplary for convenience of explanation, and the components of the electronic device (100) of the present disclosure are not limited to those shown in FIG. 1. The components of the electronic device (100) will be described in detail in FIG. 4.

[0044] The electronic device (100) may be implemented as a driver monitoring system that monitors a driver (1) inside a vehicle. In one embodiment of the present disclosure, the 'driver monitoring system (DMS)' may represent a device that captures an image of the driver (1) using a plurality of cameras (110-1 to 110-3) and recognizes the driver's (1) face from the image or recognizes the driver's (1) actions or state (e.g., drowsiness, sleep, drinking, etc.) using a pre-trained artificial intelligence model. In one embodiment of the present disclosure, the driver monitoring system may obtain biometric information from the image of the driver obtained through the plurality of cameras (110-1 to 110-3).

[0045] Referring to FIG. 1, a plurality of cameras (110-1 to 110-3) are mounted in a vehicle, and an electronic device (100) can acquire a vehicle interior image (10) through the plurality of cameras (110-1 to 110-3) (Operation 1). The electronic device (100) can acquire a plurality of vehicle interior images using the plurality of cameras (110-1 to 110-3). The plurality of vehicle interior images may be images of the driver (1).

[0046] The electronic device (100) can use a neural network model (20) to obtain a set of biometric information (30) corresponding to each of the multiple vehicle interior images and representing the biometric information of the vehicle driver from each of the multiple vehicle interior images (Operation 2). The neural network model (20) may include multiple neural network models. For example, the electronic device (100) can use the neural network model (20) to identify the driver (1) included in the vehicle interior images.

[0047] For example, the electronic device (100) can obtain heart rate information (32) of the driver (1) by using a first neural network model included in the neural network model (20). The electronic device (100) can obtain eye movement information (34) of the driver (1) by using a second neural network model included in the neural network model (20). The electronic device (100) can obtain head movement information (36) of the driver (1) by using a third neural network model included in the neural network model (20). The electronic device (100) can obtain stress information (38) of the driver (1) by using a fourth neural network model included in the neural network model (20).

[0048] The electronic device (100) can use a prompt generation model (40) to obtain a prompt set (50) including at least one prompt (52, 54, 56) that represents the driver's biometric state as text by combining at least one biometric information (32, 34, 36, 38) included in the biometric information set (30) (Operation 3.).

[0049] According to one embodiment of the present disclosure, the prompt generation model (40) may be a pre-trained language model using a plurality of biometric information set-prompt pairs as training data. The prompt generation model (40) may be a model that has learned a conversion pattern between biometric information set-prompts from a plurality of biometric information set-prompt pairs. An electronic device (100) may obtain a prompt set (50) including at least one prompt from a biometric information set (30) by using the pre-trained prompt generation model (40). In one embodiment, the prompt set (50) may include first to third prompts (52, 54, 56). The prompt generation model (40) may obtain the first to third prompts (52, 54, 56) by combining or analyzing one or more biometric information.

[0050] In one embodiment of the present disclosure, the first to third prompts (52, 54, 56) are text data generated based on the driver's biometric information and may describe or explain the driver's (1) biometric state. The first to third prompts (52, 54, 56) may express the biometric information as is (e.g., in text). For example, the first prompt may appear as "Heart rate 85 bpm, eye blinking frequency decreased by 10 beats / min." The first to third prompts (52, 54, 56) may indicate the difference between the biometric information and the biometric information corresponding to the normal driving state (e.g., rate of change or degree of relative difference). For example, the second prompt (54) may appear as "Stress index increased by 15% compared to normal state, head movement decreased by 20%." The first to third prompts (52, 54, 56) may indicate changes in the biometric information over time. For example, the third prompt (56) may appear as "The heart rate has increased by 5 compared to 1 second ago and is trending upward." The prompt generation model (40) can obtain the first to third prompts (52, 54, 56) by combining at least one biometric information, the difference between at least one biometric information and biometric information corresponding to a normal operating state, or at least one change over time of at least one biometric information.

[0051] According to one embodiment of the present disclosure, the prompt generation model (40) may be a neural network model specialized in generating a text-based prompt based on biometric information. For example, the prompt generation model (40) may identify at least one biometric signal included in the biometric information set (30) to obtain text in natural language form (e.g., a prompt).

[0052] For example, the prompt generation model (40) may be a language model (LM). More specifically, the prompt generation model (40) may be a large language model (LLM). A large language model is trained using a large dataset compared to a language model (LM) and can perform more complex language processing tasks than a language model. Since a large language model requires high-performance computing resources, it may be operated by a separate high-performance computer system (e.g., a neural network server). However, a large language model is not limited to being operated by a separate high-performance computer system, and an electronic device (100) may operate the large language model.

[0053] The electronic device (100) can obtain the driving state of the vehicle driver corresponding to the prompt set (50) using a generative artificial intelligence model (60) (Operation 4). For example, the electronic device (100) can identify whether the driving state of the vehicle driver corresponding to the prompt set (50) is a normal driving state (72), a drunk driving state (74), or a drowsy driving state (76). For example, the electronic device (100) can obtain at least one probability that the driving state of the vehicle driver corresponds to a normal driving state (72), a drunk driving state (74), or a drowsy driving state (76). The electronic device (100) can determine the driving state (70) of the vehicle driver based on each probability. In one embodiment, the driving state (70) may include at least one of a normal driving state (72), a drunk driving state (74), or a drowsy driving state (76). In one embodiment, the driving state (70) may include at least one of the probability that the driver of the vehicle is in a normal driving state (72), a drunk driving state (74), or a drowsy driving state (76).

[0054] According to one embodiment of the present disclosure, an electronic device (100) may include a generative artificial intelligence model (60). The electronic device (100) may input a prompt set (50) into the generative artificial intelligence model (60). The electronic device (100) may input the prompt set (50) and an image of the vehicle interior (10) into the generative artificial intelligence model (60). The electronic device (100) may obtain the driver's driving state (70) using the generative artificial intelligence model.

[0055] According to one embodiment of the present disclosure, the generative artificial intelligence model (60) may be a neural network model specialized in analyzing input text to generate a result of interpreting the context of the text. For example, the generative artificial intelligence model (60) may be a neural network model capable of contextually interpreting the driving state of a driver by analyzing a prompt set (50) based on a biometric information set (30). The generative artificial intelligence model (60) may be a neural network model specialized in identifying the driving state of a driver in real time by analyzing a prompt set (50) based on a biometric information set (30).

[0056] In one embodiment, a generative artificial intelligence model (60) can identify various biometric information included in a prompt set (50) and determine the driving state of the driver in an integrated manner. For example, the generative artificial intelligence model (60) can analyze the prompt set (50) to obtain at least one of the probability that the driver is in a normal driving state, the probability that the driver is in a drunk driving state, or the probability that the driver is in a drowsy driving state. For example, the generative artificial intelligence model (60) can output the driving state of the vehicle driver as "normal driving state" as the driving state (70). For example, the generative artificial intelligence model (60) can output probability values ​​corresponding to each driving state of the vehicle driver, such as "probability of normal driving: 70%", "probability of drowsy driving: 20%", and "probability of drunk driving: 10%". For example, the generative artificial intelligence model (60) can output the probability value of the vehicle driver being in a normal driving state and the probability value of being in an abnormal driving state, such as “probability of normal driving: 70%”, “probability of abnormal driving: 30%”, and “probability of drunk driving: 10%”.

[0057] For example, the generative artificial intelligence model (60) may be a language model (LM). More specifically, the generative artificial intelligence model (60) may be a large language model (LLM). The large language model is trained using a large dataset compared to the language model (LM) and can perform more complex language processing tasks than the language model. Since the large language model requires high-performance computing resources, it may be operated by a separate high-performance computer system (e.g., a neural network server). However, the large language model is not limited to being operated by a separate high-performance computer system, and the electronic device (100) may operate the large language model.

[0058] According to one embodiment of the present disclosure, the generative artificial intelligence model (60) may be a model capable of receiving and processing not only text but also images or videos. For example, the generative artificial intelligence model may be a multimodal artificial intelligence model. The multimodal artificial intelligence model may be an artificial intelligence model capable of processing various forms of data simultaneously. For example, the multimodal artificial intelligence model may be an artificial intelligence model capable of processing text data and image data together.

[0059] The electronic device (100) can identify whether the driver of the vehicle is driving normally based on the driving state of the driver corresponding to each of the multiple vehicle interior images (operation 5.).

[0060] For example, the electronic device (100) can obtain a first prompt set based on a first set of biometric information corresponding to a first vehicle interior image obtained by a first camera (110-1), and can obtain a first driving state (82) of the driver (1) based on the first prompt set. Likewise, the electronic device (100) can obtain a second driving state (84) of the driver (1) corresponding to a second vehicle interior image obtained by a second camera (110-2). Likewise, the electronic device (100) can obtain a third driving state (86) of the driver (1) corresponding to a third vehicle interior image obtained by a third camera (110-3).

[0061] The electronic device (100) can identify whether the driver of the vehicle is driving normally by comparing the first driving state (82), the second driving state (84), and the third driving state (86). For example, the electronic device (100) can identify that the driver of the vehicle is driving normally if the first driving state (82), the second driving state (84), and the third driving state (86) are all in a normal driving state. For example, if the probability that the driver of the vehicle is driving normally in the first driving state (82), the second driving state (84), and the third driving state (86) is all greater than or equal to the first threshold probability, the driver of the vehicle can identify that the driver of the vehicle is driving normally.

[0062] For example, the electronic device (100) can identify that the driver of the vehicle is not driving normally if at least one of the first driving state (82), the second driving state (84), and the third driving state (86) is not a normal driving state. For example, the electronic device (100) can identify that the driver of the vehicle is not driving normally if at least one of the probability that the driver of the vehicle is driving normally in the first driving state (82), the second driving state (84), and the third driving state (86) is less than a first threshold probability.

[0063] The electronic device (100) can output a warning based on the identification that the driver of the vehicle is not driving normally (Operation 6). For example, the electronic device (100) can output a warning message through a speaker inside or outside the vehicle or through a display inside the vehicle. For example, the electronic device (100) can transmit images acquired by the camera (110) and images acquired by the vehicle's black box to a server. For example, the electronic device (100) can control the vehicle so that the vehicle's lights output a warning light. For example, the electronic device (100) can control the vehicle to lower the vehicle's speed below a threshold speed or to perform an autonomous driving function.

[0064] The electronic device (100) can implement a highly accurate driver monitoring system by identifying whether the driver (1) is driving normally based on images acquired from multiple cameras. The electronic device (100) can improve the accuracy of driver state analysis by identifying whether the driver (1) is driving normally based on multiple prompts based on multiple biometric information.

[0065] Hereinafter, the electronic device (100) and the method of operation of the electronic device (100) will be described in detail with reference to FIGS. 2 to 15.

[0066] FIG. 2 is a block diagram illustrating the configuration of an electronic device mounted in a vehicle according to one embodiment of the present disclosure.

[0067] According to one embodiment of the present disclosure, an electronic device (100) is an electronic device mounted on a vehicle to control the vehicle, and may include a plurality of cameras (110), a processor (120), and a memory (130).

[0068] The electronic device (100) can be implemented in various forms. The electronic device (100) can be any form of device that performs functions, including a processor and memory. The electronic device (100) can be mounted or installed in a vehicle.

[0069] Multiple cameras (110) are devices that acquire images using visible light and can provide visual information through the images. Multiple cameras (110) can provide accurate images of objects through color images, and there may be no restrictions on the amount of data processed or resolution limitations due to bandwidth limitations. Multiple cameras (110) can acquire images of the interior of a vehicle equipped with an electronic device (100). For example, multiple cameras (110) can acquire images of the driver of a vehicle equipped with an electronic device (100).

[0070] A plurality of cameras (110) may be composed of cameras including a first camera (110-1), a second camera (110-2), a third camera (110-3), …, and an nth camera (110-n). Each of the plurality of cameras (110-1 to 110-n) may be positioned at different locations within the vehicle. Each of the plurality of cameras (110-1 to 110-n) may have different angles of the images they capture.

[0071] The processor (120) can control the overall operation of the electronic device (100). The processor (120) can execute one or more programs stored in the memory (130). According to one embodiment, the memory (130) can store various data, programs, or applications for driving and controlling the electronic device (100).

[0072] A processor (120) is a component that controls a series of processes to enable an electronic device (100) to operate according to the embodiments described below, and may be composed of one or more processors. One or more processors included in the processor (120) may be circuitry such as a System on Chip (SoC) or an Integrated Circuit (IC). One or more processors included in the processor (120) may be a general-purpose processor such as a CPU (Central Processing Unit), MPU (Micro Processor Unit), AP (Application Processor), or DSP (Digital Signal Processor); a graphics-dedicated processor such as a GPU (Graphic Processing Unit) or VPU (Vision Processing Unit); an artificial intelligence-dedicated processor such as a NPU (Neural Processing Unit); or a communication-dedicated processor such as a CP (Communication Processor). If one or more processors included in the processor (120) are artificial intelligence-dedicated processors, said artificial intelligence-dedicated processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0073] The processor (120) can write data to memory (130) or read data stored in memory (130), and in particular, can process data according to a predefined operation rule or artificial intelligence model by executing a program or at least one instruction stored in memory (130). Accordingly, the processor (120) can perform operations described in subsequent embodiments, and operations described as being performed by the electronic device (100) or detailed elements included in the electronic device (100) in subsequent embodiments can be seen as being performed by the processor (120) unless otherwise specified.

[0074] For example, the processor (120) may perform the function of the electronic device (100) described in the present disclosure by individually or collectively executing at least one instruction stored in the memory (130). Alternatively, according to one embodiment, the electronic device (100) may perform the function described in the present disclosure by individually or collectively executing at least one instruction stored in the memory (130) by the processor (120).

[0075] The processor (120) can process data according to a predefined operation rule or artificial intelligence model by executing a program or at least one instruction stored in memory (130). Accordingly, the processor (120) can perform operations described in subsequent embodiments, and operations described as being performed by the electronic device (100) or detailed components included in the electronic device (100) in subsequent embodiments can be seen as being performed by the processor (120) unless otherwise specified.

[0076] The memory (130) can store a program for processing and controlling the processor (120), and can store data that is input to or output from the electronic device (100). Additionally, the memory (130) can store data necessary for the operation of the electronic device (100).

[0077] The memory (130) may include at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk, and optical disk.

[0078] Memory (130) may store instructions, data structures, and program code that can be read by the processor (120). In the following embodiments, the processor (120) may be implemented by executing the instructions or code of the program stored in memory (130). Memory (130) may not exist separately but may be configured to be included in the processor (120). Memory (130) may be composed of volatile memory, non-volatile memory, or a combination of volatile and non-volatile memory. Memory (130) may store a program or at least one instruction for performing operations according to the embodiments described below. Memory (130) may provide stored data to the processor (120) upon the request of the processor (120).

[0079] By executing at least one instruction individually or collectively by at least one processor, an electronic device according to one embodiment can acquire a plurality of in-vehicle images through a plurality of cameras (110). For example, the electronic device (100) can acquire a first in-vehicle image of a vehicle (1000) through a first camera (110-1). For example, the electronic device (100) can acquire a second in-vehicle image of a vehicle (1000) through a second camera (110-2). For example, the electronic device (100) can acquire a third in-vehicle image of a vehicle through a third camera (110-3).

[0080] By executing at least one instruction individually or collectively by at least one processor, an electronic device according to one embodiment can acquire a set of biometric information corresponding to each of a plurality of vehicle interior images and representing biometric information of a vehicle driver from each of a plurality of vehicle interior images using a plurality of neural network models. The set of biometric information may include at least one of heart rate information, eye movement information, head movement information, or stress information. By executing at least one instruction individually or collectively by at least one processor, an electronic device according to one embodiment can acquire heart rate information from a first neural network model. By executing at least one instruction individually or collectively by at least one processor, an electronic device according to one embodiment can acquire eye movement information from a second neural network model. By executing at least one instruction individually or collectively by at least one processor, an electronic device according to one embodiment can acquire head movement information from a third neural network model. By executing at least one instruction individually or collectively by at least one processor, an electronic device according to one embodiment can obtain stress information from a fourth neural network model. A specific embodiment in which the electronic device (100) obtains a set of bio-information from each of a plurality of vehicle interior images will be described in detail with reference to FIGS. 5a and 5b.

[0081] By executing at least one instruction individually or collectively by at least one processor, an electronic device according to one embodiment can obtain a prompt set including at least one prompt that represents the driver's biometric state as text by combining at least one biometric information included in a set of biometric information using a prompt generation model. A specific embodiment in which the electronic device (100) obtains at least one prompt using a prompt generation model will be described in detail with reference to FIGS. 6a and 6b.

[0082] By executing at least one instruction individually or collectively by at least one processor, an electronic device according to one embodiment can obtain a driving state of a vehicle driver corresponding to a prompt set using a generative artificial intelligence model. The driving state may include a normal driving state, a drowsy driving state, or a drunk driving state. Specific embodiments of how the electronic device (100) obtains the driving state of a vehicle driver will be described in detail with reference to FIGS. 7a and 7b.

[0083] By having at least one instruction executed individually or collectively by at least one processor, an electronic device according to one embodiment can identify whether the driver of a vehicle is driving normally based on the driving state of the driver corresponding to each of a plurality of vehicle interior images. A specific embodiment of how the electronic device (100) identifies whether the driver of a vehicle is driving normally will be described in detail with reference to FIGS. 8a and 8b.

[0084] By executing at least one instruction individually or collectively by at least one processor, an electronic device according to one embodiment may output a warning based on the identification that the driver of the vehicle is not in normal driving. For example, the electronic device may further include a display (140). The electronic device (100) may output a warning through the display (140). For example, the electronic device may further include a speaker (150). The electronic device (100) may output a warning through the speaker (150). Specific embodiments of how the electronic device (100) outputs a warning will be described in detail with reference to FIG. 10 and FIG. 11a through FIG. 11d.

[0085] Although not illustrated in FIG. 2, the electronic device (100) can control various devices of the vehicle (e.g., engine control system, autonomous driving system, headlights, or taillights, etc.) through an Electronic Control Unit (ECU). In one embodiment of the present disclosure, the processor (120) of the electronic device (100) can perform functions and / or operations to protect the driver and people around the vehicle by controlling various devices of the vehicle through the ECU when abnormal breathing of the driver is detected, thereby lowering the driving speed, performing autonomous driving, and outputting a warning light. A specific embodiment in which the electronic device (100) controls the devices of the vehicle through the ECU when the driver is identified as not being in normal driving is to be described in detail in FIG. 11.

[0086] In the disclosed embodiment, at least one of the operations performed by the processor (120) may be performed using artificial intelligence (AI) technology. At least one operation performed using artificial intelligence (AI) technology is described in detail below with reference to FIG. 12.

[0087] FIG. 3 is a flowchart illustrating a method for controlling a vehicle through an electronic device according to one embodiment of the present disclosure.

[0088] Referring to FIG. 3, in operation 310, an electronic device (100) mounted in a vehicle can acquire multiple in-vehicle images through multiple cameras (110). The multiple cameras (110) may be mounted at different positions or angles within the vehicle. The electronic device (100) can acquire in-vehicle images including the driver from multiple different perspectives through the multiple cameras (110).

[0089] In operation 320, the electronic device (100) can obtain a set of biometric information corresponding to each of a plurality of vehicle interior images and representing biometric information of the vehicle driver. In one embodiment, the electronic device (100) can obtain a set of biometric information corresponding to each of a plurality of vehicle interior images and representing biometric information of the vehicle driver from each of a plurality of vehicle interior images using a plurality of neural network models.

[0090] For example, the electronic device (100) may acquire a first set of biometric information including at least one of the first heart rate information, first eye movement information, first head movement information, or first stress information, which are biometric information of a driver included in the first internal image. For example, the electronic device (100) may acquire the first heart rate information using a first neural network model, acquire the first eye movement information using a second neural network model, acquire the first head movement information using a third neural network model, and acquire the first stress information using a fourth neural network model. Operation 320 will be described in detail with reference to FIGS. 5a, 5b, and 5c.

[0091] In operation 330, the electronic device (100) can use a prompt generation model to obtain at least one prompt that represents the driver's biometric state as text by combining at least one biometric information. In one embodiment, the electronic device (100) can use a prompt generation model to obtain a set of prompts that includes at least one prompt that represents the driver's biometric state as text by combining at least one biometric information included in a set of biometric information.

[0092] For example, the electronic device (100) can obtain at least one prompt by combining at least one biometric data (or, measured value, absolute value). For example, the electronic device (100) can obtain at least one prompt by combining at least one biometric data and the difference (or, relative value, comparison value) of biometric information corresponding to a normal operating state. For example, the electronic device (100) can obtain at least one prompt by combining at least one biometric information and the difference between the biometric information and the biometric information corresponding to a normal operating state. Operation 330 will be described in detail with reference to FIGS. 6a and FIGS. 6b.

[0093] In operation 340, the electronic device (100) can obtain the driving state of the vehicle driver corresponding to the prompt set by using a generative artificial intelligence model. For example, the electronic device (100) can identify each biometric information included in the prompt set and the relationship between the biometric information by using a generative artificial intelligence model. The driving state may include at least one of the probability that the vehicle driver is in a normal driving state, the probability that the driver is in a drowsy driving state, or the probability that the driver is in a drunk driving state. The electronic device (100) can obtain at least one of the probability that the vehicle driver is in a normal driving state, the probability that the driver is in a drowsy driving state, or the probability that the driver is in a drunk driving state. Operation 340 will be described in detail with reference to FIGS. 7a and 7b.

[0094] In operation 350, the electronic device (100) can identify whether the driver of the vehicle is driving normally based on the driving state of the driver corresponding to each of the plurality of vehicle interior images. For example, the electronic device (100) can identify that the driver of the vehicle is driving normally if all of the probabilities that the driver of the vehicle is in a normal driving state included in the driving state of the driver corresponding to each of the plurality of vehicle interior images are greater than or equal to the first threshold probability. For example, the electronic device (100) can identify that the driver of the vehicle is not driving normally if at least one of the probabilities that the driver of the vehicle is in a normal driving state included in the driving state of the driver corresponding to each of the plurality of vehicle interior images is less than the first threshold probability. Operation 350 will be described in detail with reference to FIGS. 8a and FIGS. 8b.

[0095] In operation 360, the electronic device (100) may output a warning based on the identification that the driver of the vehicle is not driving normally. For example, the electronic device (100) may output a warning message through at least one of a display or a speaker. For example, the electronic device (100) may transmit video of the vehicle interior or black box video to a server through a communication interface. For example, the electronic device (100) may control the vehicle to output a warning light through the vehicle's lights (e.g., headlights, taillights). For example, the electronic device (100) may control the vehicle to lower the vehicle's driving speed below a threshold speed or to activate an autonomous driving function, and detailed operations will be described in detail with reference to FIG. 10 and FIG. 11a through FIG. 11d.

[0096] FIG. 4 is a reference diagram for explaining a method in which an electronic device acquires multiple vehicle interior images through a plurality of cameras according to one embodiment of the present disclosure.

[0097] In one embodiment of the present disclosure, an electronic device (100) can acquire multiple images of the interior of a vehicle through a plurality of cameras. Each of the plurality of cameras may have a unique shooting position and angle. Each of the plurality of cameras may capture different parts, such as the driver's face, upper body, and hand movements. For example, among the plurality of cameras, a front dashboard camera may capture a frontal image of the driver, a ceiling camera may capture the driver's head and upper body from a top-down view, and a rear camera may capture the driver's side or rear view.

[0098] In one embodiment of the present disclosure, the electronic device (100) can acquire a first interior image (410) of the vehicle through a first camera (110-1) among a plurality of cameras. For example, the first interior image (410) may be an image of the driver (1) taken from the right side. In one embodiment, the electronic device (100) can acquire a second interior image (420) of the vehicle through a second camera (110-2) among a plurality of cameras. For example, the second interior image (420) may be an image of the driver (1) taken from the front. In one embodiment, the electronic device (100) can acquire a third interior image (430) of the vehicle through a third camera (110-3) among a plurality of cameras. For example, the third interior image (430) may be an image of the driver (1) taken from the right ceiling.

[0099] In one embodiment of the present disclosure, the electronic device (100) can identify the driver's biometric information from multiple angles by analyzing internal images taken at multiple angles. For example, the electronic device (100) can obtain the driver's (1) heart rate information by identifying changes in facial blood flow or skin tone based on a second internal image (420) taken from the front. For example, the electronic device (100) can obtain the driver's (1) head movement information by identifying changes in head movement based on a second internal image (420) taken from the front. For example, the electronic device (100) can obtain the driver's (1) head movement information by identifying changes in the driver's (1) head movement based on a third internal image (430) taken from the right ceiling.

[0100] The electronic device (100) can acquire biometric information of the driver (1) from various viewpoints through multiple vehicle interior images captured at multiple angles. The electronic device (100) can increase the accuracy of the acquired biometric information by acquiring the driver's (1) biometric information based on the vehicle interior images captured at various angles. The electronic device (100) can increase the reliability of identifying the driver's (1) driving state results based on the biometric information. In addition, the data captured by each camera can play a complementary role. For example, the second camera (110-2) can acquire biometric information that the first camera (110-1) cannot acquire, and the electronic device (100) can determine the driver's driving state based on the biometric information acquired by the first camera (110-1) and the second camera (110-2).

[0101] The electronic device (100) improves the accuracy of acquiring the driver's biometric information by utilizing image data from various angles, and can identify abnormal conditions of the driver, such as drowsiness, alcohol consumption, and abnormal heart rate, more quickly and accurately. By using multiple cameras, the electronic device (100) reduces data loss (e.g., blind spots) or analysis errors that may occur when using a single camera, and enables stable monitoring of the driver's condition even in various environments inside the vehicle (e.g., lighting conditions, changes in the driver's position, etc.).

[0102] The number and placement positions of the plurality of cameras (110-1 to 110-3) shown in FIG. 4 are exemplary for convenience of explanation, and the components of the electronic device (100) of the present disclosure are not limited to those shown in FIG. 4. For example, the plurality of cameras (110-1 to 110-3) may be installed on at least one of the ceiling, dashboard, steering wheel, steering column, glove box, overhead console, sun visor, and rearview mirror of the vehicle (1000), and are not limited to the described embodiments.

[0103] FIG. 5a is a flowchart illustrating a method for an electronic device included in operation 320 according to one embodiment of the present disclosure to acquire a set of bio-information using a plurality of neural network models.

[0104] Referring to operation 322, the electronic device (100) can acquire a first set of biometric information of a vehicle driver. In one embodiment, the electronic device (100) can acquire a first set of biometric information of a vehicle driver based on a first interior image of the vehicle acquired by a first camera (110-1) using a plurality of neural network models. The first set of biometric information may include first heart rate information, first eye movement information, first head movement information, and first stress index information.

[0105] FIG. 5b is a reference diagram illustrating a method for an electronic device to acquire a first set of bio-information based on a first internal image using a plurality of neural network models according to one embodiment of the present disclosure.

[0106] Referring to FIG. 5b, the electronic device (100) can input a first internal image (410) to a plurality of neural network models (510). The plurality of neural network models may include at least one of a first neural network model, a second neural network model, a third neural network model, or a fourth neural network model.

[0107] In one embodiment, the electronic device (100) may undergo a preprocessing process before inputting the first interior image (410) into a plurality of neural network models. For example, the electronic device (100) may adjust the brightness of the first interior image (410) according to the vehicle interior brightness conditions. For example, the electronic device (100) may improve the quality of the image by removing noise contained in the first interior image (410).

[0108] The electronic device (100) can input a first internal image (410) into a plurality of neural network models (510) to obtain a first set of biometric information (520) from the plurality of neural network models (510). In one embodiment, the electronic device (100) can obtain first heart rate information (520-1) based on the first internal image (410) using the first neural network model (510-1). For example, the electronic device (100) can identify a driver from the first internal image (410) using the first neural network model (510-1). The electronic device (100) can identify a change in skin color in the face area of ​​the driver included in the first internal image (410) based on the first neural network model (510-1). The electronic device (100) can obtain the driver's (1) heart rate through frequency analysis based on identifying that a change in skin color in the face area is caused by blood circulation using the first neural network model (510-1), but the method of obtaining the driver's heart rate is not limited to the described embodiment. In one embodiment, the first neural network model (510-1) may be trained to obtain heart rate information based on an image containing a person. In one embodiment, the first neural network model (510-1) may be stored in the electronic device (100).

[0109] In one embodiment, an electronic device (100) can obtain first eye movement information (520-2) based on a first internal image (410) using a second neural network model (510-2). For example, the second neural network model (510-2) can assume the screen of the first internal image (410) as a single two-dimensional coordinate plane and identify eye movements on the two-dimensional coordinate plane. The electronic device (100) can identify a driver's face region included in the first internal image (410) using the second neural network model (510-2). The electronic device (100) can set a bounding box on the driver's face region included in the first internal image (410) using the second neural network model (510-2) to set a Region of Interest (ROI). The electronic device (100) can identify the center coordinates of the driver's left eye and right eye included in the first internal image (410) using the second neural network model (510-2). The electronic device (100) can identify the center coordinates of the driver's left eye after a certain time included in the first internal image (410) using the second neural network model (510-2).

[0110] The electronic device (100) can identify at least one of the speed of eye movement or the frequency of eye movement during a reference time (e.g., 50ms, 1 second, 3 seconds, 1 minute) by using the second neural network model (510-2) to identify changes in the driver's eye center coordinates (e.g., left eye center coordinates or right eye center coordinates) included in the first internal image (410). The electronic device (100) can identify the angle of eye movement during a reference time (e.g., 50ms, 1 second, 3 seconds, 1 minute) by using the second neural network model (510-2) to identify changes in the driver's eye center coordinates (e.g., left eye center coordinates or right eye center coordinates) included in the first internal image (410). However, the method of obtaining driver's eye movement information is not limited to the described embodiments.

[0111] In one embodiment, the second neural network model (510-2) may be trained to acquire eye movement information based on an image containing a person. In one embodiment, the second neural network model (510-2) may be stored in the electronic device (100).

[0112] In one embodiment, an electronic device (100) can obtain first head movement information (520-3) based on a first internal image (410) using a third neural network model (510-3). For example, the third neural network model (510-3) can identify head movements by assuming the screen of the first internal image (410) as a single two-dimensional coordinate plane. The electronic device (100) can identify the driver's head region included in the first internal image (410) using the third neural network model (510-3). The electronic device (100) can identify the driver's head center coordinates included in the first internal image (410) using the third neural network model (510-3). The electronic device (100) can identify the driver's head center coordinates after a certain time included in the first internal image (410) using the third neural network model (510-3).

[0113] The third neural network model (510-3) can identify changes in the center coordinates of the driver's head included in the first internal image (410) to identify at least one of the head movement speed, head movement frequency, or head movement angle range during a reference time (e.g., 50 ms, 1 second, 3 seconds, 1 minute). The electronic device (100) can identify the head rotation angle based on the first internal image (410) using the third neural network model (510-3). For example, the third neural network model (510-3) can perform 3D pose estimation techniques. However, the method of obtaining driver's head movement information is not limited to the described embodiments.

[0114] In one embodiment, the third neural network model (510-3) may be trained to acquire head movement information based on an image containing a person. In one embodiment, the third neural network model (510-3) may be stored in the electronic device (100).

[0115] In one embodiment, the electronic device (100) may obtain first stress information (520-4) based on the first internal image (410) using the fourth neural network model (510-4). For example, the electronic device (100) may obtain a driver's stress index based on at least one of the driver's facial expression, the driver's pupil size, the driver's eye blinking degree, or the driver's heart rate included in the first internal image (410) using the fourth neural network model (510-4). However, the method of obtaining the driver's stress information is not limited to the described embodiment.

[0116] In one embodiment, the fourth neural network model (510-4) may be trained to acquire stress information based on an image containing a person. In one embodiment, the fourth neural network model (510-4) may be stored in the electronic device (100).

[0117] The electronic device (100) can input a first set of biometric information (520) corresponding to a first internal image (410) into a prompt generation model (530) to obtain a first prompt set (540). The specific process by which the electronic device (100) obtains the first prompt set will be explained in detail with reference to FIGS. 6a and 6b.

[0118] The number of multiple neural network models (510-1 to 510-4) shown in FIG. 5b is exemplary for convenience of explanation, and the components of the neural network models included in the electronic device (100) of the present disclosure are not limited to those shown in FIG. 5b. For example, the multiple neural network models (510) may include a fifth neural network model trained to acquire hand movement information based on an image containing a person. The electronic device (100) may acquire first hand movement information based on a first internal image (410) using the fifth neural network model.

[0119] The plurality of biometric information (heart rate information, eye movement information, head movement information, stress information) illustrated in FIG. 5b are exemplary for convenience of explanation, and the biometric information included in the first set of biometric information is not limited to the examples described above and may further include other biometric information obtained from the first internal image.

[0120] The first, second, third, and fourth neural network models are terms used to distinguish them from one another. The first, second, third, or fourth neural network models do not necessarily have to be different neural network models. The first, second, third, and fourth neural network models may be the same or different neural network models. The first, second, third, and fourth neural network models may also be a single neural network model.

[0121] FIG. 5c is a reference diagram showing a first set of bio-information according to one embodiment of the present disclosure.

[0122] The electronic device (100) can acquire a first set of biometric information (520) based on a first internal image (410) using a plurality of neural network models. In one embodiment, the first set of biometric information (520) may include a plurality of biometric information of the driver (1) of the vehicle (1000).

[0123] For example, the first set of biometric information (520) may include first heart rate information. The heart rate information may be expressed in units of beats per minute (bpm) (e.g., 112 bpm).

[0124] For example, the first set of biometric information (520) may include first eye movement information. The first eye movement information may include a change in the position coordinates of the driver (1) of the vehicle's eye (or pupil) within the first internal image (410) per reference time (e.g., moving from (3,5) to (2,4) in 1 second). The first eye movement information may include an eye movement speed determined based on the change in the position coordinates of the eye (or pupil) (e.g., moving 2 units in 1 second). The first eye movement information may include an eye movement frequency. For example, the first eye movement information may include the number of blinks per reference time (e.g., 1 second) of the pupil (e.g., 6 blinks in 1 second). The reference time may be determined according to the frame rate of each of the plurality of cameras.

[0125] For example, the first set of biometric information (520) may include first head movement information. The first head movement information may be represented as a position change value per reference time (e.g., 1 second) of head position coordinates (e.g., head center coordinates) (e.g., moving from (7,3) to (5,2) in 1 second). The first head movement information may include a head movement speed determined based on the change in head position coordinates (e.g., moving 5 units in 1 second). The first head movement information may include a head movement frequency. For example, the first head movement information may include the number of eye movements per reference time (e.g., 1 second) of the pupils (e.g., moving 4 times in 1 second). The first head movement information may be represented as an angle change per reference time (e.g., 1 second) of head center coordinates (e.g., moving 5 degrees to the left and right in 1 second). The reference time may be determined according to the frame rate of each of the plurality of cameras.

[0126] For example, the first set of bio-information (520) may include first stress information. The stress information may be represented as a stress index (e.g., 60, 60%, 0.6).

[0127] For convenience of explanation, FIG. 5c illustrates that the first biometric information set (520) includes heart rate information, eye movement information, head movement information, and stress information, but the embodiments of the present disclosure are not limited thereto. For example, the first biometric information set (520) may not include at least one of heart rate information, eye movement information, head movement information, and stress information. For example, the first biometric information set (520) may include other biometric information of the driver other than heart rate information, eye movement information, head movement information, and stress information.

[0128] Returning to FIG. 5a and referring to operation 324, the electronic device (100) can acquire a second set of biometric information of the vehicle driver. In one embodiment, the electronic device (100) can acquire a second set of biometric information of the vehicle driver based on a second interior image of the vehicle acquired by a second camera (110-2) using a plurality of neural network models. The second set of biometric information may include second heart rate information, second eye movement information, second head movement information, and second stress information. Since the second set of biometric information acquired by the electronic device (100) in operation 324 is identical to the first set of biometric information described with reference to operation 322 and FIG. 5b and FIG. 5c, except that it is based on the second interior image (420), redundant description is omitted.

[0129] Referring to operation 326, the electronic device (100) can acquire a third set of biometric information of the vehicle driver. In one embodiment, the electronic device (100) can acquire a third set of biometric information of the vehicle driver based on a third interior image of the vehicle acquired by a third camera (110-3) using a plurality of neural network models. The third set of biometric information may include third heart rate information, third eye movement information, third head movement information, and third stress information. Since the third set of biometric information acquired by the electronic device (100) in operation 324 is identical to the first set of biometric information described with reference to operation 322 and FIG. 5b and FIG. 5c, except that it is based on the third interior image (430), redundant description is omitted.

[0130] In one embodiment of the present disclosure, a plurality of sets of biometric information may include different types of biometric information. For example, the types of biometric information included in a first set of biometric information, a second set of biometric information, and a third set of biometric information may be different from each other. For example, the first set of biometric information may include first heart rate information and first eye movement information, the second set of biometric information may include second eye movement information, second head movement information, and second stress information, and the third set of biometric information may include third heart rate information, third head movement information, and third stress information.

[0131] The electronic device (100) can compensate for biometric information that cannot be obtained from a specific angle or has low accuracy when obtained from a specific angle by obtaining biometric information that cannot be obtained from a specific angle of camera in a different angle of camera in a video. The electronic device (100) can improve accuracy when determining the driver's driving state based on biometric information by obtaining the same type of biometric information multiple times from multiple videos.

[0132] FIG. 6a is a flowchart illustrating a method for an electronic device included in operation 330 according to one embodiment of the present disclosure to obtain a prompt using a prompt generation model.

[0133] In one embodiment, the electronic device (100) may acquire biometric information corresponding to a normal driving state. The electronic device (100) may store biometric information corresponding to a normal driving state, or may receive biometric information corresponding to a normal driving state from a server or an external device. The electronic device (100) may store biometric information corresponding to a normal driving state that is personalized to the driver (1). For example, the electronic device (100) may include at least one of heart rate information corresponding to a normal driving state, eye movement information corresponding to a normal driving state, head movement information corresponding to a normal driving state, and stress information corresponding to a normal driving state.

[0134] Hereinafter, heart rate information corresponding to normal driving conditions will be referred to as 'normal heart rate information', eye movement information corresponding to normal driving conditions as 'normal eye movement information', head movement information corresponding to normal driving conditions as 'normal head movement information', and stress information corresponding to normal driving conditions as 'normal stress information'.

[0135] Referring to operation 332, the electronic device (100) can identify the difference between at least one biometric information included in the biometric information set and biometric information corresponding to a normal operating state. For example, the electronic device (100) can identify the difference between at least one biometric information value included in the biometric information set and a range of biometric information corresponding to a normal operating state. For example, the electronic device (100) can identify the difference between the change trend (or rate of change) of at least one biometric information included in the biometric information set and the change trend (or rate of change) of biometric information corresponding to a normal operating state.

[0136] For example, the electronic device (100) can identify the difference between the first heart rate information included in the first set of biometric information and the normal heart rate information. For example, if the first heart rate information is 112 bpm and the normal heart rate range is 70 bpm to 90 bpm, the electronic device (100) can identify the difference between the first heart rate information and the normal heart rate information as "112 bpm - 90 bpm (normal maximum) = 22 bpm higher than the normal heart rate." For example, if the first heart rate information is trending upward by 5 bpm per second and the normal heart rate change trend is within 2 bpm per second, the electronic device (100) can identify the difference between the first heart rate information and the normal heart rate information as "3 bpm / sec faster than the normal heart rate change trend." The normal heart rate information may be the normal heart rate range of a typical adult, or it may be a normal heart rate range personalized for the driver (1).

[0137] For example, the electronic device (100) can identify the difference between the first eye movement information included in the first set of biometric information and the normal eye movement information. For example, if the first eye movement information is "5 blinks per minute, pupils move within a 10-degree range to the left and right per second" and the normal eye movement range is 10 to 20 blinks per minute, coordinate movement within a 30-degree range to the left and right, the electronic device (100) can identify the difference between the first eye movement information and the normal eye movement information as 10 times (normal minimum) - 5 times = 5 times or within a 30-degree range = no difference. The normal eye movement information may be the normal eye movement range of a typical adult, or it may be a normal eye movement range personalized for the driver (1).

[0138] For example, the electronic device (100) can identify the difference between the first head movement information included in the first set of biometric information and the normal head movement information. For example, if the first head movement information is "movement within a 5-degree range to the left and right of the head center for 1 second" and the normal head movement range is "movement within a 15-degree range to the left and right," the electronic device (100) can identify the difference between the first head movement information and the normal head movement information as "within a 15-degree range = no difference." The normal head movement information may be the normal head movement range of a typical adult, or it may be a normal head movement range personalized for the driver (1).

[0139] For example, the electronic device (100) can identify the difference between the first stress information included in the first set of bio-information and the normal stress information. For example, if the first stress information is "stress index 60" and the normal stress index range is 30 to 50, the electronic device (100) can identify the difference between the first stress information and the normal stress information as 60 - 50 (maximum value of the normal range) = 10. The normal stress information may be the normal stress index range of a typical adult, or it may be a stress index range personalized for the driver (1).

[0140] Referring to operation 334, the electronic device (100) can obtain at least one prompt based on the difference between at least one biometric information included in a set of biometric information or biometric information corresponding to a normal operating state using a prompt generation model. In one embodiment, the prompt generation model may be a model trained to generate a prompt representing a biometric state by combining a plurality of biometric information or the difference between a plurality of biometric information and biometric information corresponding to a normal operating state. In one embodiment, the prompt generation model may be a model trained to generate a prompt representing a set of biometric information by interpreting each biometric information and combining the set of biometric information into various expressions. The prompt generation model may be a model that has learned a conversion pattern between a set of biometric information and a prompt from a plurality of biometric information set-prompt pairs.

[0141] In one embodiment, the prompt generation model may receive as input a prompt set including at least one biometric information obtained from an image inside a vehicle. In one embodiment, the prompt generation model may receive as input a prompt set including at least one biometric information obtained in real-time from an image inside a vehicle. In one embodiment, the prompt generation model may receive as input a prompt set including at least a change in biometric information obtained from an image inside a vehicle.

[0142] In one embodiment, the prompt generation model can generate a sentence indicating the driver's driving state by integrating (or combining) at least one biometric information included in the prompt set. For example, the prompt generation model can generate text listing at least one biometric information included in the prompt set. For example, the prompt generation model can generate text containing information interpreting the difference between at least one biometric information included in the prompt set and biometric information corresponding to a normal state. The electronic device (100) can obtain a prompt from a set of biometric information using a pre-trained prompt generation model. The electronic device (100) can obtain a plurality of prompts from a set of biometric information using a pre-trained prompt generation model.

[0143] In one embodiment, the electronic device (100) can obtain a first prompt set based on a first set of biometric information corresponding to a first internal image using a prompt generation model. The electronic device (100) can obtain a second prompt set based on a second set of biometric information corresponding to a second internal image using a prompt generation model. The electronic device (100) can obtain a third prompt set based on a third set of biometric information corresponding to a third internal image using a prompt generation model.

[0144] The electronic device (100) can obtain prompts that express the driver's biometric state in various ways by obtaining multiple prompts corresponding to a set of biometric information. When the electronic device (100) determines the driver's state based on multiple prompts, it can improve the accuracy of the driver's state analysis compared to when it determines the driver's state using a generative artificial intelligence model based on a single prompt.

[0145] FIG. 6b is a reference diagram for illustrating a prompt set according to one embodiment of the present disclosure.

[0146] The electronic device (100) can obtain a first prompt set (620) based on a first set of biometric information (610) using a prompt generation model. The electronic device (100) can obtain a first prompt set (620) containing at least one prompt by inputting the first set of biometric information (610) into the prompt generation model.

[0147] In one embodiment, the first heart rate information included in the first bio-information set (610) may be 112 bpm, the first eye movement information may be moving from (3,5) to (2,4) for 1 second, the first head movement information may be moving from (7,3) to (5,2) for 1 second, and the first stress information may be 60.

[0148] In one embodiment, the first heart rate information is 112 bpm, and the difference between the heart rate information corresponding to a normal driving state and the first heart rate information may be a 24% increase, 22 bpm, or a 22 bpm increase.

[0149] In one embodiment, the first eye movement information may be represented as moving from (3,5) to (2,4) in 1 second, moving 2 units in 1 second, and moving within ±10 degrees in 1 second. The difference between the eye movement information corresponding to the normal driving state and the first eye movement information may be represented as a 70% reduction in eye movement and movement within the normal eye movement range.

[0150] In one embodiment, the first head movement information may be represented as moving from (7,3) to (5,2), moving 3 units per second, and moving within ±5 degrees per second. The difference between the head movement information corresponding to the normal driving state and the first head movement information may be represented as a 40% reduction in head movement and movement within the normal head movement range.

[0151] In one embodiment, the first stress information is 60, and the difference between the stress information corresponding to the normal operating state and the first stress information may be a 20% increase or a 10 increase.

[0152] The electronic device (100) can obtain at least one prompt based on the difference between at least one biometric information included in a set of biometric information or biometric information corresponding to a normal operating state using a prompt generation model.

[0153] For example, the electronic device (100) can obtain a first prompt using a prompt generation model, and the first prompt may be "HR increased by 22, eye movement decreased by 70%, head movement decreased by 40%, stress increased by 20%". For example, the electronic device (100) can obtain a second prompt using a prompt generation model, and the second prompt may be "HR 112, eye movement 2 times / sec, head coordinates (7,3) to (5,2), stress index 60". For example, the electronic device (100) can obtain a third prompt using a prompt generation model, and the third prompt may be "HR increased by 22, eye coordinates (3,5) to (2,4) for 1 second, head movement less than ±5 degrees, stress increased by 20%".

[0154] The electronic device (100) obtains a prompt based on the difference between 'at least one' biometric information or biometric information corresponding to a normal operating state using a prompt generation model, so that at least one biometric information included in the biometric information set may be excluded from the prompt. For example, the prompt generation model may obtain a prompt based on the remaining biometric information without selectively using some of the biometric information within the biometric information set.

[0155] For example, the electronic device (100) can obtain a fourth prompt based on first heart rate information, first eye movement information, and first stress information. For example, the electronic device (100) can obtain a fifth prompt based on first head movement information and first stress information.

[0156] In one embodiment, the electronic device (100) can obtain a larger number of prompts by obtaining prompts in which some biometric information within the biometric information set is not selectively used. In one embodiment, the electronic device (100) can exclude some biometric information with low accuracy from prompt generation by obtaining prompts in which some biometric information within the biometric information set is not selectively used. For example, eye movement information included in a biometric information set based on an internal image taken from the side may have low accuracy. When the electronic device (100) obtains prompts based on biometric information included in a biometric information set based on an internal image taken from the side, it can generate prompts by combining the remaining biometric information excluding the eye movement information.

[0157] Although the first set of bio-information according to one embodiment is described in FIG. 6b, the second set of bio-information is identical to the first set of bio-information except that it is based on the second internal image (420), and the third set of bio-information is identical to the first set of bio-information except that it is based on the third internal image (430), so redundant descriptions are omitted.

[0158] FIG. 7a is a flowchart for explaining in detail a method for obtaining the driving state of a vehicle driver corresponding to each of a plurality of vehicle interior images included in operation 340 according to one embodiment of the present disclosure.

[0159] Referring to operation 342, the electronic device (100) can acquire a driving state of a vehicle driver corresponding to each of a plurality of vehicle interior images. The driving state may include a normal driving state, a drowsy driving state, and a drunk driving state. The driving state may be determined based on at least one of the probability that the vehicle driver is in a normal driving state, a drowsy driving state, or a drunk driving state.

[0160] In one embodiment, the electronic device (100) can identify whether the driver of a vehicle corresponding to each of a plurality of vehicle interior images is in a normal driving state, a drowsy driving state, or a drunk driving state. In one embodiment, the electronic device (100) can obtain at least one probability that the driver of a vehicle corresponding to each of a plurality of vehicle interior images is in a normal driving state, a drowsy driving state, or a drunk driving state. In one embodiment, the electronic device (100) can obtain the driving state of the driver of a vehicle corresponding to each of a plurality of vehicle interior images by inputting a prompt set corresponding to each of a plurality of vehicle interior images into a generative artificial intelligence model.

[0161] For example, the electronic device (100) can obtain a first driving state of a driver of a vehicle corresponding to a first internal image by inputting a first prompt set into a generative artificial intelligence model. For example, the first driving state may include at least one of the probability that the driver of the vehicle corresponding to the first internal image is in a normal driving state, a drowsy driving state, or a drunk driving state. The generative artificial intelligence model can identify the driving state of a driver corresponding to the first prompt set based on a combination of biometric information included in the input first prompt set.

[0162] In one embodiment, the electronic device (100) can analyze a first prompt set using a generative artificial intelligence model. The generative artificial intelligence model can individually analyze the biometric information included in the input first prompt set and combine them into a single context to extract a pattern of each biometric information data (e.g., increase / decrease ratio, correlation, etc.). For example, the generative artificial intelligence model can infer that if the heart rate increases and eye movements become sluggish, there is a high probability of being in a drunk driving state.

[0163] The electronic device (100) can infer the driver's state based on data learned from relationships and patterns between biometric information using a generative artificial intelligence model. The generative artificial intelligence model may be a model that has been pre-trained based on a combination of biometric information and a large amount of biometric data (e.g., at least one of text data, numeric data, image data, and video data) regarding normal driving state, drowsy driving state, and drunk driving state. The generative artificial intelligence model can identify the driver's driving state by comparing the features of an input prompt with the learned data and identifying the most matching pattern.

[0164] For example, a generative AI model can identify whether a driver is in a normal driving state, a drowsy driving state, or a drunk driving state. For example, a generative AI model can identify at least one of the probability that the driver is in a normal driving state, the probability that the driver is in a drowsy driving state, or the probability that the driver is in a drunk driving state. For example, a generative AI model can identify only the probability that the driver is in a normal driving state and the probability that the driver is not in a normal driving state. For example, a generative AI model can identify all of the probability that the driver is in a normal driving state, the probability that the driver is in a drowsy driving state, or the probability that the driver is in a drunk driving state.

[0165] Likewise, the electronic device (100) can obtain a second driving state of the vehicle driver corresponding to a second internal image by inputting a second prompt set into a generative artificial intelligence model. The electronic device (100) can obtain a third driving state of the vehicle driver corresponding to a third internal image by inputting a third prompt set into a generative artificial intelligence model, and since the second driving state and the third driving state are identical to the first driving state except that they are based on the second prompt set and the third prompt set, redundant descriptions are omitted.

[0166] FIG. 7b is a flowchart for explaining in detail a method for obtaining the driving state of a vehicle driver corresponding to each of a plurality of vehicle interior images included in operation 340 according to one embodiment of the present disclosure.

[0167] Referring to operation 344, the electronic device (100) can obtain a driving state of a vehicle driver corresponding to each of a plurality of vehicle interior images. The driving state may include at least one of a normal driving state, a drowsy driving state, and a drunk driving state. Each driving state may be determined based on probability. In one embodiment, the electronic device (100) can obtain at least one probability that the vehicle driver corresponding to each of a plurality of vehicle interior images is in a normal driving state, a drowsy driving state, or a drunk driving state. In one embodiment, the electronic device (100) can obtain a probability that the vehicle driver corresponding to each of a plurality of vehicle interior images is in at least one of the driving states by inputting each of the plurality of vehicle interior images and a prompt set corresponding to each of the plurality of vehicle interior images into a generative artificial intelligence model.

[0168] For example, the electronic device (100) can obtain a first driving state of a driver of a vehicle corresponding to a first internal image by inputting a first prompt set and a first internal image into a generative artificial intelligence model. For example, the first driving state may include at least one of the probability that the driver of the vehicle corresponding to the first internal image is in a normal driving state, a drowsy driving state, or a drunk driving state.

[0169] A generative artificial intelligence model can identify the driving state of a driver corresponding to a first prompt set based on a combination of biometric information included in an input first prompt set and a first internal image. The generative artificial intelligence model can accurately infer the driving state by simultaneously receiving a first prompt set, which is text data, and a first internal image, which is image data. The generative artificial intelligence model may be a multimodal artificial intelligence model.

[0170] In one embodiment, the electronic device (100) can analyze a first prompt set using a generative artificial intelligence model. Since the process of the electronic device (100) analyzing the first prompt set is identical to the operation described earlier with reference to operation 742 in FIG. 7a, redundant descriptions will be omitted for brevity.

[0171] In one embodiment, a generative AI model can determine the driving state of a driver by considering a first prompt set and a first internal image together. For example, the generative AI model can identify patterns of summarized biometric information or changes in biometric state from the first prompt set and obtain additional biometric information not included in the prompt (e.g., subtle posture changes, subtle movements) from the first internal image. The generative AI model can improve the accuracy of inference by obtaining the driving state of the driver by combining text data (e.g., prompt set) and image data (e.g., internal image).

[0172] A generative AI model can identify at least one of the probability that the driver is in a normal driving state, a drowsy driving state, or a drunk driving state. For example, a generative AI model can identify only the probability that the driver is in a normal driving state and the probability that the driver is not in a normal driving state. For example, a generative AI model can identify all of the probability that the driver is in a normal driving state, a drowsy driving state, or a drunk driving state.

[0173] In one embodiment, the electronic device (100) can obtain a second driving state of a vehicle driver corresponding to a second internal image by inputting a second prompt set and a second internal image into a generative artificial intelligence model. The electronic device (100) can obtain a third driving state of a vehicle driver corresponding to a third internal image by inputting a third prompt set and a third internal image into a generative artificial intelligence model, and since the second driving state and the third driving state are identical to the first driving state except that they are based on the second prompt set and the third prompt set, redundant descriptions are omitted.

[0174] FIGS. 8A and 8B are drawings for explaining in detail an operation in which an electronic device included in operation 350 according to one embodiment of the present disclosure identifies whether the driver of a vehicle is driving normally based on the driving state of the driver corresponding to each of a plurality of vehicle interior images.

[0175] Hereinafter, with reference to FIG. 8a and FIG. 8b together, the function and / or operation of the electronic device (100) identifying whether the driver of the vehicle is driving normally will be described in detail.

[0176] Referring to operation 352, the electronic device (100) can identify the driving state of a vehicle driver corresponding to each of a plurality of vehicle interior images. In one embodiment, the electronic device (100) can identify whether the probability that the driver included in the driving state is in a normal driving state is all greater than or equal to a first threshold probability. In one embodiment, if the driving state of a vehicle driver corresponding to each of a plurality of vehicle interior images is all in a normal driving state, the vehicle driver can be identified as being in normal driving.

[0177] Referring to operation 354, if the driving state of the vehicle driver corresponding to each of the plurality of vehicle interior images is a normal driving state, the electronic device (100) can determine that the vehicle driver is driving normally. Referring to the electronic device (100), if the probability that the vehicle driver is in a normal driving state included in the driving state of the vehicle driver corresponding to each of the plurality of vehicle interior images is all greater than or equal to a first threshold probability, the electronic device (100) can determine that the vehicle driver is driving normally. The first threshold probability may be a value stored in the electronic device (100).

[0178] For example, referring to FIG. 8b, in the first embodiment (810), the first threshold probability may be 70%. The electronic device (100) may obtain an 80% probability that the driver of the vehicle included in the driving state of the vehicle corresponding to the first internal image obtained by the first camera (110-1) included in the electronic device (100) is in a normal driving state, obtain an 85% probability that the driver of the vehicle included in the driving state of the vehicle corresponding to the second internal image obtained by the second camera (110-2) is in a normal driving state, and obtain a 75% probability that the driver of the vehicle included in the driving state of the vehicle corresponding to the third internal image obtained by the third camera (110-3) is in a normal driving state. The electronic device (100) may determine that the driver of the vehicle is in a normal driving state based on the fact that the probability that the driver of the vehicle included in the driving state of the vehicle corresponding to each of the first to third internal images is in a normal driving state is all greater than or equal to the first threshold probability. The above-described embodiment is a case where the number of cameras included in the electronic device (100) is three, and the present disclosure is not limited thereto.

[0179] Returning to FIG. 8a and referring to operation 356, if at least one of the probability that the driver of the vehicle is in a normal driving state, which is included in the driving state of the driver of the vehicle corresponding to each of the multiple vehicle interior images, is less than the first threshold probability, the electronic device (100) can determine that the driver of the vehicle is not in a normal driving state.

[0180] For example, referring to FIG. 8b, in the second embodiment (820), the electronic device (100) can obtain a probability of 65% that the driver of the vehicle included in the first driving state, which is the driving state of the vehicle corresponding to the first internal image, is in a normal driving state. The electronic device (100) can determine that the driver of the vehicle is not in a normal driving state because at least one of the probabilities that the driver of the vehicle included in the first driving state, the second driving state, and the third driving state is in a normal driving state is less than the first threshold probability.

[0181] In one embodiment, if the probability that the driver of the vehicle is in a drowsy driving state included in the driving state of the driver of the vehicle corresponding to each of the plurality of vehicle interior images is all greater than or equal to a second threshold probability, the electronic device (100) can determine that the driver of the vehicle is drowsy driving.

[0182] For example, referring to FIG. 8b, in the second embodiment (820), the second threshold probability may be 75%. The electronic device (100) may obtain a probability of 90% that the driver of the vehicle included in the first driving state is in a drowsy driving state, a probability of 85% that the driver of the vehicle included in the second driving state is in a drowsy driving state, and a probability of 80% that the driver of the vehicle included in the third driving state is in a drowsy driving state. Since the probability that the driver of the vehicle included in the first driving state, the second driving state, and the third driving state is in a drowsy driving state is all greater than or equal to the second threshold probability, the electronic device (100) may determine that the driver of the vehicle is not driving normally.

[0183] If the electronic device (100) determines that the driver of the vehicle is in a drunk driving state included in the driving state of the driver of the vehicle corresponding to each of the multiple vehicle interior images, the electronic device (100) can determine that the driver of the vehicle is drunk driving.

[0184] For example, referring to FIG. 8b, in the third embodiment (830), the second threshold probability may be 80%. The electronic device (100) may obtain a probability of 90% that the driver of the vehicle included in the first driving state is in a drunk driving state, a probability of 85% that the driver of the vehicle included in the second driving state is in a drunk driving state, and a probability of 80% that the driver of the vehicle included in the third driving state is in a drunk driving state. Since the probability that the driver of the vehicle included in the first driving state, the second driving state, and the third driving state is in a drunk driving state is all greater than or equal to the third threshold probability, the electronic device (100) may determine that the driver of the vehicle is not driving normally.

[0185] The first critical probability, second critical probability, and third critical probability are terms used to distinguish them from one another. The first, second, and third critical probabilities do not necessarily have to be different probabilities. The first, second, and third critical probabilities may be the same or different probabilities.

[0186] Returning to FIG. 8a and referring to operation 360, the electronic device (100) can output a warning. In one embodiment, the electronic device (100) can output a warning if it identifies that the driver of the vehicle is not driving normally. The electronic device (100) can output a warning in various ways. An embodiment in which the electronic device (100) outputs a warning will be described in detail with reference to FIG. 10 and FIG. 11a through FIG. 11d, and for brevity, redundant descriptions are omitted here.

[0187] FIG. 9 is a flowchart illustrating an operation in which an electronic device determines whether the heart rate of a vehicle driver falls within a normal range prior to operation 310 or operation 360 according to one embodiment of the present disclosure.

[0188] Referring to operation 910, the electronic device (100) can identify whether the driver's heart rate of the vehicle falls within the normal range when the vehicle (1000) is turned on.

[0189] Referring to operation 912, the electronic device (100) can acquire the heart rate of the vehicle driver (1) based on the vehicle's ignition being turned on. In one embodiment, based on the vehicle's ignition being turned on, the electronic device (100) can acquire an image of the vehicle interior with at least one of a plurality of cameras (110). In one embodiment, the electronic device (100) can acquire an image of the vehicle interior by selecting any camera among the plurality of cameras (110). For example, the electronic device (100) can acquire an image of the vehicle interior with at least one of a first camera (110-1), a second camera (110-2), or a third camera (110-3). Based on the acquired image of the vehicle interior, the electronic device (100) can acquire the heart rate of the vehicle driver using a first neural network model. The detailed operation of obtaining the heart rate of the driver of the vehicle (1) using a first neural network model based on the vehicle interior image obtained by the electronic device (100) corresponds to the operation of obtaining the heart rate of the driver of the vehicle described above with reference to FIGS. 5a to 5c, so for brevity, the redundant explanation is omitted here.

[0190] In one embodiment, based on the vehicle's ignition being turned on, the electronic device (100) can obtain the heart rate of the vehicle's driver (1) through an external device (not shown). In one embodiment, the electronic device (100) can obtain the heart rate through the vehicle's driver (1)'s user device (e.g., a smartphone, a smartwatch, a smart ring, etc.). For example, the electronic device (100) can receive real-time heart rate data of the driver (1) by communicating with a user device such as a smartwatch, a smartphone, or a smart ring.

[0191] Referring to operation 914, the electronic device (100) can identify whether the heart rate of the vehicle's driver falls within a normal range. For example, the normal heart rate range may be 60 bpm or higher and 100 bpm, which is the typical normal heart rate range for an adult. For example, the normal heart rate range may also be defined by criteria individualized according to the driver. For example, the electronic device (100) can set a driver-specific normal heart rate range by learning average heart rate data recorded by the vehicle's driver during past driving.

[0192] In one embodiment, the electronic device (100) may perform operation 310 if the heart rate of the vehicle driver falls within a normal range. In one embodiment, the electronic device (100) may perform operation 360 and output a warning if the heart rate of the vehicle driver does not fall within a normal range. For example, the electronic device (100) may perform operation 360 if the heart rate of the vehicle driver is higher than the normal range. For example, the electronic device (100) may perform operation 360 if the heart rate of the vehicle driver is lower than the normal range. For example, the electronic device (100) may perform operation 360 if it cannot determine (or decide) whether the heart rate of the vehicle driver falls within a normal range. For example, the electronic device (100) may perform operation 360 if the heart rate trend of the vehicle driver does not fall within a normal range. For example, the electronic device (100) can determine that the driver's heart rate of the vehicle is not within the normal range if it is rapidly increasing or rapidly decreasing, and perform operation 360.

[0193] In one embodiment of the present disclosure, in operation 910, the electronic device (100) can determine whether the driver (1) is in a state to drive normally before or at the time of starting to drive by identifying whether the driver's heart rate falls within a normal range when the vehicle (1000) is turned on. The electronic device (100) can significantly improve the safety of the driver and the surrounding environment by determining that the driver is in a state to drive normally if the driver's heart rate falls within a normal range and outputting a warning, thereby protecting the driver (1), other drivers, and pedestrians, and preventing road traffic accidents.

[0194] FIG. 10 is a flowchart illustrating an operation in which an electronic device included in operation 360 according to one embodiment of the present disclosure outputs a warning based on the identification that the driver of the vehicle is not in normal driving.

[0195] The electronic device (100) may perform operation 360 after operation 350, and may also perform operation 360 even if it is identified in operation 914 that the driver's heart rate of the vehicle is not within the normal range, as previously explained with reference to FIG. 9.

[0196] Referring to operation 362 of FIG. 10, the electronic device (100) may output a warning message. For example, the electronic device (100) may output a warning message indicating that the driver (1) is not driving normally if the difference between the initial heart rate and the normal heart rate does not fall within a threshold range. The electronic device (100) may output a warning message indicating that the driver (1) is not driving normally based on at least one of the driver's driving condition being identified as drowsy driving or drunk driving.

[0197] For example, the electronic device (100) can output a warning message through a display of a vehicle. In one embodiment, the electronic device (100) further includes a display unit (140, see FIG. 15) and can display a warning message through the display (140). For example, the electronic device (100) can output a warning message through a speaker inside or outside the vehicle. In one embodiment, the electronic device (100) further includes a speaker (150, see FIG. 15) and can output a warning message through the speaker (150). The electronic device (100) can control at least one of the display unit (140) or the speaker (150) to output a warning message through at least one of the display unit (140) or the speaker (150). Specific embodiments of the electronic device (100) outputting a warning message are to be described with reference to FIG. 11a through FIG. 11d, and for brevity, redundant descriptions are omitted here.

[0198] Referring to operation 364, the electronic device (100) may transmit at least one of a plurality of vehicle interior images or images of the vehicle exterior obtained through the vehicle's black box to a server. In one embodiment, the electronic device (100) may transmit at least one of a plurality of vehicle interior images or images of the vehicle exterior obtained through the vehicle's black box to a server if the difference between the initial heart rate and the normal heart rate is not included in a threshold range. The electronic device (100) may transmit at least one of a plurality of vehicle interior images or images of the vehicle exterior obtained through the vehicle's black box to a server based on at least one of the driver being identified as drowsy driving or drunk driving. The electronic device (100) further includes a communication interface (160, see FIG. 15) and may control the communication interface (160) to transmit at least one of a plurality of vehicle interior images or images of the vehicle exterior obtained through the vehicle's black box to a server through the communication interface (160). In one embodiment, the electronic device (100) may transmit a captured image in real time and may transmit a stored image.

[0199] In one embodiment, a plurality of vehicle interior images may be images of the driver (1). In one embodiment, a plurality of vehicle interior images may indicate whether the driver (1) is driving normally or driving abnormally (e.g., drowsy driving, drunk driving, or in a state of unconsciousness). In one embodiment, an image of the vehicle outside obtained through the vehicle's black box may be an image of surrounding road conditions, the movement of other vehicles, or interactions with pedestrians while driving.

[0200] For example, the electronic device (100) can transmit video of the vehicle interior containing a driver who is not driving normally, or video of the vehicle exterior obtained through the vehicle's black box, to an emergency rescue agency (e.g., fire department, medical institution, etc.) via a communication interface (160). By transmitting video of the vehicle interior or video of the vehicle exterior obtained through the vehicle's black box to the emergency rescue agency, the electronic device (100) can support the emergency rescue agency in quickly grasping the situation and efficiently carrying out rescue activities.

[0201] For example, the electronic device (100) further includes a communication interface (160, see FIG. 15) and can transmit video of the vehicle interior containing a driver who is not driving normally, or video of the vehicle exterior obtained through the vehicle's black box, to a traffic accident-related agency (e.g., police station, insurance company, etc.) through the communication interface (160). By transmitting video of the vehicle interior or video of the vehicle exterior obtained through the vehicle's black box to a traffic accident-related agency, the electronic device (100) can provide objective evidence for the analysis of the cause of the accident and the distribution of responsibility when a traffic accident occurs.

[0202] The electronic device (100) can improve responsiveness to emergency situations, prevent traffic accidents, and support post-accident measures by effectively utilizing internal and external vehicle image data through operation 364.

[0203] Referring to operation 366, the electronic device (100) can control the vehicle (1000) so that the vehicle's lights output a warning light. In one embodiment, the electronic device (100) can control the vehicle (1000) so that the vehicle's lights (e.g., headlights, taillights) output a warning light if the difference between the initial heart rate and the normal heart rate is not within a threshold range. In one embodiment, the electronic device (100) can control the vehicle (1000) so that the vehicle's lights (e.g., headlights, taillights) output a warning light based on at least one of the driver being identified as drowsy driving or drunk driving.

[0204] For example, the electronic device (100) can send a signal to a lighting control system (1040, see FIG. 12) to generate a control command to cause the vehicle's lights to flash (e.g., periodic flashing) or output a warning light in a specific pattern. For example, the pattern of the warning light can be set to red or yellow to indicate an emergency situation. For example, the pattern of the warning light can be set so that other vehicles and pedestrians can easily recognize it while driving.

[0205] The electronic device (100) can control the vehicle's headlights (1042, see FIG. 12) and taillights (1044, see FIG. 12) individually or simultaneously. For example, the electronic device (100) can transmit a signal to a lighting control system (1040) to cause the headlights (1042) to blink at 1-second intervals and the taillights (1044) to perform continuous flashing to provide a warning to following vehicles. By controlling the vehicle (1000) so that the vehicle's lights output a warning light, the electronic device (100) can prevent accidents by quickly notifying surrounding vehicles and pedestrians of the driver's abnormal condition and support rescue activities in emergency situations.

[0206] Referring to operation 368, the electronic device (100) can control the vehicle (1000) to control the vehicle's operating speed to below a threshold speed or activate the vehicle's autonomous driving function if the difference between the initial heart rate and the normal heart rate does not fall within a threshold range. The electronic device (100) can control the vehicle (1000) to control the vehicle's operating speed to below a threshold speed or activate the vehicle's autonomous driving function based on at least one of the driver being identified as drowsy driving or drunk driving.

[0207] In one embodiment, the electronic device (100) can control the vehicle's operating speed to be below a threshold speed. For example, the electronic device (100) can send a signal to the vehicle's engine control system (1020, see FIG. 12) to limit the vehicle speed to below a threshold speed (e.g., 30 km / h). When the electronic device (100) sends a signal to the vehicle's engine control unit (ECU), the vehicle's brake system and accelerator pedal control can be synchronized to maintain the speed so as not to exceed the set speed.

[0208] In one embodiment, the electronic device (100) can activate the autonomous driving function of the vehicle. For example, the electronic device (100) can send a command to the vehicle's autonomous driving system (1030, see FIG. 12) to activate an autonomous driving mode so that the vehicle can move safely without driver intervention. When the autonomous driving function is activated, the vehicle (1000) can drive by using sensors such as GPS, LiDAR, and cameras to recognize the road environment and select a safe route. For example, the vehicle can move to a nearby parking lot and stop, or guide the vehicle to the nearest rest area when driving on a highway.

[0209] The electronic device (100) provides a technical effect that protects the driver and prevents traffic accidents by lowering the driving speed of the vehicle (1000) and activating the autonomous driving function when abnormal driving by the driver (e.g., drowsy driving, drunk driving) is detected and the driver is unable to drive normally.

[0210] In one embodiment, the operations described above with reference to operations 362 through 368 may be performed individually or simultaneously. Additionally, operations 362 through 368 may or may not be performed in order.

[0211] For example, if the difference between the initial heart rate and the normal heart rate is not within the threshold range or if the driver's driving condition is identified as drowsy or intoxicated, the electronic device (100) can activate a speed limit or autonomous driving function according to operation 368, and at the same time provide a warning message to the driver through the vehicle's internal speaker according to operation 362, or visually display relevant information on the vehicle's display.

[0212] For example, the electronic device (100) can control the vehicle (1000) so that the vehicle's lights emit a warning light before outputting a warning message if the difference between the initial heart rate and the normal heart rate does not fall within a threshold range or if the driver's driving condition is identified as drowsy or intoxicated.

[0213] FIGS. 11a to 11d are reference drawings for explaining the operation of an electronic device outputting a warning according to one embodiment of the present disclosure.

[0214] Referring to FIG. 11a, an electronic device (100) can output (or display) a warning message (1110) through a display (140a) in a vehicle. In one embodiment of the present disclosure, the warning message (1110) may include at least one of whether the driver of the vehicle is drowsy or drunk driving, a warning notification, or a behavioral guide. In one embodiment of the present disclosure, the display (140a) is implemented as a center information display (CID) in a vehicle, and the electronic device (100) can display the warning message (1110) through the CID.

[0215] In the embodiment illustrated in FIG. 11a, an electronic device (100) mounted in a vehicle may display a notification message (1100) indicating calibration information on a CID. The warning message (1110) may consist of text indicating, for example, "Drowsy driving warning! Take a break." The warning message (1110) may consist of text indicating, for example, "Drunk driving warning! Stop driving. Driving after drinking threatens your own life and the lives of others." However, the warning message is not limited to the warning message (1110) illustrated in FIG. 11a.

[0216] FIG. 11b is a diagram illustrating the operation of an electronic device (100) according to one embodiment of the present disclosure displaying a warning message (1120).

[0217] Referring to FIG. 11b, the electronic device (100) can output (or display) a warning message (1120) through a Head Up Display (HUD) (140b) projected onto the windshield (1130) of the vehicle. In the embodiment illustrated in FIG. 11b, the electronic device (100) may further include a projector (1132) in addition to the components illustrated in FIG. 2. The projector (1132) is an optical engine configured to project an image onto the windshield (1130), which is the front windshield of the vehicle. The projector (1132) is configured to generate light for the image and may be an optical engine including an image panel, a lighting optical system, a projection optical system, etc. In one embodiment of the present disclosure, a projector (1132) may acquire text data or graphic data (or image data) constituting a warning message (1120) from a processor (120, see FIG. 2), generate a virtual image based on the acquired text data or graphic data (or image data), and project light constituting the virtual image output from a light source onto a windshield (1130) through an emission surface. The image projected by the projector (1132) may be displayed through a HUD (140b) on the windshield (1130).

[0218] The warning message (1120) is information intended to stop the driver from driving while drowsy or drunk and to protect the driver by providing information warning that the driver is driving while drowsy or drunk and providing a guide for action to stop driving while drowsy or drunk. The warning message (1120) shown in FIG. 11b is identical to the warning message (1110) shown in FIG. 11a except that it is displayed via the HUD (140b), so a redundant description is omitted.

[0219] Referring to FIG. 11c, the electronic device (100) can output (or display) a warning message (1140) on the instrument display (140c) of the vehicle. The warning message (1140) is information intended to stop drowsy driving or drunk driving and to protect the driver by providing information (1142) warning that the driver is drowsy driving or drunk driving and a guide to action (1144) to stop drowsy driving or drunk driving.

[0220] When driving a vehicle, the driver receives approximately 80% to 90% of situational awareness information through their eyes. The electronic device (100) according to the embodiment illustrated in FIG. 11a, 11b, and 11c outputs warning messages (1110, 1120, 1140) through the vehicle's display (140a), HUD (140b), or instrument panel display (140c) so that they can be easily viewed without interfering with driving, thereby inducing a driver who is drowsy or drunk to stop driving. Through this, the electronic device (100) according to one embodiment of the present disclosure provides a technical effect that can stop the driver's abnormal driving and prevent the occurrence of road traffic accidents.

[0221] Referring to FIG. 11d, the electronic device (100) can output (or play, transmit) a warning message (1150) through the vehicle's speakers (150-1, 150-2). The warning message (1150) may include voice containing information warning the driver that they are drowsy or drunk driving, and a guide for action to stop drowsy or drunk driving. For example, the warning message (1150) may include voice reading, "Drowsy driving warning! Please take a break. The nearest rest area is on the right after 1 km," but is not limited thereto. The electronic device (100) can output the warning message (1150) through at least one speaker (150-1, 150-2) inside the vehicle. The warning message (1150) can also be output through at least one speaker (not shown) outside the vehicle.

[0222] When a driver is driving under the influence of alcohol or while drowsy, the driver may find it difficult to properly observe the road ahead. The electronic device (100) according to the embodiment illustrated in FIG. 11d outputs a warning message (1150) through a speaker inside the vehicle, thereby inducing the driver, who is driving abnormally and unable to observe the road ahead, to stop driving. The electronic device (100) according to the embodiment illustrated in FIG. 11d outputs a warning message (1150) through a speaker outside the vehicle, thereby allowing other drivers or pedestrians around the vehicle (1000) to drive or walk with caution regarding the vehicle (1000) being driven by a driver who is driving while drowsy or under the influence of alcohol. Through this, the electronic device (100) according to one embodiment of the present disclosure provides a technical effect that can stop the driver's abnormal driving and prevent the occurrence of road traffic accidents.

[0223] The warnings output by the electronic device are not limited to the warning messages (1110, 1120, 1140, 1150) shown in FIGS. 11a through 11d above. The electronic device (100) may output the warning through at least one of, for example, a navigation device, a passenger seat display, and a user device (for example, a smartphone, tablet, smartwatch, or smart ring).

[0224] FIG. 12 is a block diagram illustrating the operation of controlling devices included in a vehicle as an electronic device according to one embodiment of the present disclosure identifies that the driver is not driving normally.

[0225] Referring to FIG. 12, the vehicle (1000) may include an ECU (1010), an engine control system (1020), an autonomous driving system (1030), and a lighting control system (1040). The lighting control system (1040) may include headlights (1042) and taillights (1044). FIG. 12 only illustrates configurations for explaining the operation of controlling the devices of the vehicle (1000) as the electronic device (100) identifies that the driver is not driving normally, and the devices included in the vehicle (1000) are not limited to those shown in FIG. 12.

[0226] An ECU (electronic control unit) (1010) is an electronic device that controls the functions and / or operations of devices within a vehicle, such as a braking system and a steering system, as well as powertrains such as the vehicle's engine and automatic transmission. A connector (C) is a hardware device that connects the electronic device (100) and devices within the vehicle (1000). In one embodiment of the present disclosure, the electronic device (100) may be physically and / or electrically interconnected with the ECU (1010) through the connector (C). The electronic device (100) may transmit control signals to the ECU (1010) through the connector (C). In one embodiment, the electronic device (100) may transmit control signals to the ECU (1010) via a wireless communication connection.

[0227] According to one embodiment of the present disclosure, the electronic device (100) can protect the driver by transmitting a control signal to the ECU (1010) upon identifying that the driver is not driving normally, and by controlling various devices of the vehicle through the ECU (1010) to control the speed of the vehicle and perform autonomous driving. According to one embodiment of the present disclosure, the processor (120, see FIG. 2) of the electronic device (100) can protect other drivers or pedestrians by transmitting a control signal to the ECU (1010) upon identifying that the driver is not driving normally, and by controlling various devices of the vehicle through the ECU (1010) to output a warning outside the vehicle.

[0228] According to one embodiment of the present disclosure, as it is identified that the driver is not driving normally, the electronic device (100) can control the engine control system (1020) through the ECU (1010) to reduce the speed of the vehicle. For example, the electronic device (100) can control the engine control system (1020) through the ECU (1010) to reduce the speed of the vehicle to a critical speed (e.g., 30 km / h) or lower. For example, the electronic device (100) can reduce the speed of the vehicle by controlling the throttle valve within the engine control system (1020) through the ECU (1010) to reduce the amount of air intake. For example, the electronic device (100) can reduce the speed of the vehicle by controlling the engine control system (1020) through the ECU (1010) to reduce the amount of fuel injection. For example, the electronic device (100) can reduce the speed of the vehicle by controlling the engine control system (1020) through the ECU (1010) to switch to a low-speed gear and activate the brake system.

[0229] According to one embodiment of the present disclosure, when it is identified that the driver is not driving normally, the electronic device (100) can control the autonomous driving system (1030) through the ECU (1010) to activate the autonomous driving function of the vehicle. For example, the electronic device (100) can control the autonomous driving system (1030) through the ECU (1010) to activate the autonomous driving function of the vehicle in stages. For example, the electronic device (100) can control the autonomous driving system (1030) through the ECU (1010) to activate a higher level of autonomous driving function over time.

[0230] According to one embodiment of the present disclosure, when it is identified that the driver is not driving normally, the electronic device (100) can control the lighting control system (1040) through the ECU (1010) to output a warning. For example, the electronic device (100) can control at least one of the headlight (1042) or the taillight (1044) through the ECU (1010) to output a warning light that flashes at a short period. The electronic device (100) can transmit a warning signal to vehicles and pedestrians in front and behind by outputting a warning light that flashes at a short period through at least one of the headlight (1042) or the taillight (1044). In the above-described embodiment, it was stated that the headlight (1042) or taillight (1044) outputs a warning light that 'blinks at a short interval,' but the interval and form of the warning light according to the present disclosure are not limited to the above-described examples.

[0231] FIG. 13 is a diagram illustrating an operation performed using artificial intelligence technology in a disclosed embodiment.

[0232] Specifically, at least one of the following operations performed in the electronic device (100) can be performed using artificial intelligence (AI) technology that performs computations through a neural network: i) obtaining a set of biometric information corresponding to each of each of the multiple vehicle interior images and representing biometric information of the driver of the vehicle from each of the multiple vehicle interior images; ii) obtaining a set of prompts including at least one prompt representing the biometric state of the driver as text by combining at least one biometric information included in the set of biometric information using a prompt generation model; and iii) obtaining the driving state of the driver of the vehicle corresponding to the set of prompts using a generative artificial intelligence model.

[0233] Artificial intelligence technology (hereinafter referred to as 'AI technology') is a technology that obtains a desired result by performing computations through neural networks to process input data, such as analyzing and / or classifying it.

[0234] Such AI technology can be implemented by utilizing algorithms. Here, an algorithm or a set of algorithms for implementing AI technology is referred to as a neural network. A neural network receives input data, performs the aforementioned operations for analysis and / or classification, and can output result data. In order for a neural network to accurately output result data corresponding to the input data, it is necessary to train the neural network. Here, 'training' refers to inputting various data into the neural network and training the network so that it can independently discover or learn methods for analyzing the input data, methods for classifying the input data, and / or methods for extracting features necessary for generating result data from the input data. Specifically, through the training process, the neural network can train on training data (e.g., multiple different images) to optimize and set the weight values ​​within the neural network. Then, by independently learning the input data through a neural network equipped with optimized weight values, it outputs the desired result.

[0235] Specifically, a neural network can be classified as a deep neural network when the number of hidden layers, which are internal layers that perform computations, is multiple—that is, when the depth of the neural network performing computations increases. Examples of neural networks include, but are not limited to, CNN (Convolutional Neural Network), DNN (Deep Neural Network), RNN (Recurrent Neural Network), RBM (Restricted Boltzmann Machine), DBN (Deep Belief Network), BRDNN (Bidirectional Recurrent Deep Neural Network), and Deep Q-Networks. Additionally, neural networks can be subdivided. For example, a CNN neural network can be subdivided into a DCNN (Deep Convolutional Neural Network) or a Capsnet neural network (not shown).

[0236] In the disclosed embodiments, the 'AI model' may refer to a neural network comprising at least one layer that operates to receive input data and output a desired result. Additionally, the 'AI model' may refer to an algorithm or a set of multiple algorithms that performs operations through a neural network to output a desired result, a processor for executing such algorithm or the set thereof, software for executing such algorithm or the set thereof, or hardware for executing such algorithm or the set thereof.

[0237] Referring to FIG. 13, a neural network (1300) can be trained by receiving training data. Then, the trained neural network (1300) receives input data (1310) through an input terminal (1320), and the input terminal (1320), hidden layer (1330), and output terminal (1340) can perform operations to analyze the input data (1310) and output the desired result, which is output data (1350). The operations through the neural network can be performed through the hidden layer (1330). In FIG. 10, for convenience, the hidden layer (1330) is simplified to be formed as a single layer, but the hidden layer (1330) can be formed as multiple layers.

[0238] Specifically, in the disclosed embodiment, the neural network (1300) may be trained to acquire a set of biometric information representing the biometric information of the vehicle's driver from an image inside the vehicle. Additionally, in the disclosed embodiment, the neural network (1300) may be trained to acquire a set of prompts including at least one prompt by combining at least one set of biometric information. Additionally, in the disclosed embodiment, the neural network (1300) may be trained to acquire the driving state of the vehicle's driver corresponding to the set of prompts.

[0239] In the disclosed embodiment, a neural network (1300) that performs at least one of the following operations can be implemented in a processor (120) of an electronic device (100): i) acquiring a set of biometric information corresponding to each of the plurality of vehicle interior images and representing biometric information of the vehicle driver from each of the aforementioned plurality of vehicle interior images; ii) acquiring a set of prompts including at least one prompt representing the biometric state of the driver as text by combining at least one biometric information included in the set of biometric information using a prompt generation model; and iii) acquiring the driving state of the vehicle driver corresponding to the set of prompts using a generative artificial intelligence model. The processor (120) may include an artificial intelligence dedicated processor such as a Neural Processing Unit (NPU).

[0240] Alternatively, a neural network (1300) that performs at least one of the following operations may be distinguished from the electronic device (100) and may be implemented in a separate electronic device (not shown) or processor (not shown) located within the vehicle.

[0241] In addition, the computation through the aforementioned neural network may be performed on a server (not shown) capable of communicating via a wireless communication network with an electronic device (100) mounted on a vehicle according to one embodiment. Communication between the electronic device (100) and the server (not shown) will be described in detail below with reference to FIGS. 14 and FIGS. 15.

[0242] FIG. 14 is a drawing illustrating a disclosed embodiment in which the electronic device of the present disclosure operates in conjunction with a server.

[0243] The server (1400) may include a server, server system, server-based device, etc. that transmits and receives data with an electronic device (100) and processes data through a communication network (1401).

[0244] In the disclosed embodiment, the server (1400) includes a communication interface that communicates with an electronic device (100) mounted on a vehicle (1000), and a processor that performs at least one instruction.

[0245] The server (1400) can receive information on the surrounding state of the vehicle, an image of the surrounding state of the vehicle, and information on the state of an object located around the vehicle from the electronic device (100).

[0246] The server (1400) may train an AI model and store the trained AI model. The server (1400) may use the trained AI model and received information to perform at least one of the following operations: i) obtaining a set of biometric information corresponding to each of each of the multiple vehicle interior images and representing biometric information of the driver of the vehicle from each of the multiple vehicle interior images; ii) obtaining a set of prompts including at least one prompt representing the biometric state of the driver as text by combining at least one biometric information included in the set of biometric information using a prompt generation model; and iii) obtaining the driving state of the driver of the vehicle corresponding to the set of prompts using a generative artificial intelligence model.

[0247] Generally, the electronic device (100) may have limited memory storage capacity, processing speed of computation, and ability to collect training data sets compared to the server (1400). Therefore, operations requiring the storage of large amounts of data and large amounts of computation can be performed on the server (1400), and then the necessary data and / or AI models used can be transmitted to the electronic device (100) mounted on the vehicle via a communication network. Then, the electronic device (100) mounted on the vehicle can perform the necessary operations quickly and easily by receiving and using the necessary data and / or AI models through the server, without a processor having large memory and fast computation capabilities.

[0248] In the disclosed embodiment, the server (1400) may include the neural network (1300) described in FIG. 13.

[0249] FIG. 15 is a drawing for explaining FIG. 14 in detail.

[0250] Referring to FIG. 15, the electronic device (100) may include a plurality of cameras (110), a processor (120), a memory (130), a display (140), a speaker (150), a communication interface (160), and a black box (170). Since the plurality of cameras (110), the processor (120), and the memory (130) of the electronic device (100) have been described in detail with reference to FIG. 2, they will be omitted here.

[0251] The display (140) may be configured to display a warning message under the control of the processor (120). For example, the display (140) may be implemented as an instrument cluster display. For example, the display (140) may be implemented as a Head Up Display (HUD). For example, the display (140) may be configured as at least one of a Center Information Display (CID), a navigation device, or a passenger seat display.

[0252] The display (140) may include a screen composed of at least one of, for example, a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode display, a flexible display, a 3D display, or an electrophoretic display.

[0253] A speaker (150) is a device configured to output an acoustic signal. The speaker (150) may output a warning message indicating abnormal driving under the control of a processor (120). The warning message may include, for example, "Drowsy driving warning! Take a break." The warning message may include, for example, "Drunk driving warning! Stop driving." In one embodiment of the present disclosure, the speaker (150) may output a warning sound under the control of the processor (120) when it is identified that the driver is not driving normally. The speaker (150) may be composed of a plurality of speakers including a first speaker (150-1), a second speaker (150-2), a third speaker (150-3), …, and an nth speaker (150-n). Each of the plurality of speakers (150-1 to 150-n) may be placed at different locations within the vehicle.

[0254] The communication interface (160) can communicate with at least one electronic device. Here, 'communication' may mean the operation of transmitting and / or receiving data, signals, requests, and / or commands, etc. The communication interface (160) can perform wired or wireless communication with at least one electronic device. The electronic device (100) can communicate with the server (1400) through the communication interface (160).

[0255] For example, the communication interface (160) may include at least one of a communication module, a communication circuit, a communication device, an input / output port, and an input / output plug for performing wired or wireless communication with at least one electronic device. For example, the communication interface (160) may include at least one wireless communication module, a wireless communication circuit, or a wireless communication device for performing wireless communication with at least one electronic device.

[0256] For example, the communication interface (160) may include a short-range communication module, such as an IR (infrared) communication module, capable of receiving control commands from a remote controller located at a short distance. In this case, the communication interface (160) may receive control signals from the remote controller.

[0257] For example, the communication interface (160) may include at least one communication module that performs communication according to wireless communication standards such as Bluetooth, Wi-Fi, BLE (Bluetooth Low Energy), NFC (Near Field Communication), RFID (Radio Frequency Identification), Wi-Fi Direct, UWB, or Zigbee. Alternatively, the communication interface (160) may further include a communication module that performs communication with a server to support long-distance communication according to long-distance communication standards. For example, the communication interface (160) may include a communication module that performs communication through a network for internet communication. Additionally, the communication interface (160) may include a hardware communication device that performs data communication with a base station, a server, or other devices around the vehicle through a communication network according to communication standards such as 3G, 4G, 5G and / or 6G.

[0258] The black box (170) is a type of drive video record system (DVRS) that captures images of the surroundings of the vehicle (1000) during driving, parking, and / or stopping, and stores the acquired images. The black box (170) detects events occurring during driving, parking, or stopping of the vehicle (1000), such as damage to the vehicle (1000) or theft of items inside the vehicle (1000), such as contact accidents or rear-end collisions, and can acquire images by capturing the surroundings of the vehicle (1000) as the event is detected.

[0259] The communication interface (1410) of the server (1400) may include one or more components that enable communication with the electronic device (100). The communication interface (1410) includes at least one communication module, such as a short-range communication module, a wired communication module, a mobile communication module, a broadcast reception module, etc. Here, the at least one communication module refers to a communication module capable of transmitting and receiving data through a network that follows a communication standard, such as a tuner that performs broadcast reception, Bluetooth, WLAN (Wireless LAN) (Wi-Fi), Wibro (Wireless broadband), WiMAX (World Interoperability for Microwave Access), CDMA, WCDMA, the Internet, 3G, 4G, and / or 5G, or a method of performing communication using millimeter wave (mmWAVE).

[0260] For example, if the communication interface (1410) performs communication using millimeter wave (mmWAVE), large amounts of data can be transmitted and received quickly. Specifically, by rapidly receiving large amounts of data in a vehicle, data necessary for vehicle safety (e.g., data necessary for autonomous driving, data necessary for navigation services, etc.) and user content (e.g., movies, music, etc.) can be rapidly provided, thereby increasing the safety of the vehicle and / or the convenience of the user.

[0261] Specifically, the mobile communication module included in the communication interface (1410) can communicate with another device located at a distance (e.g., a server (not shown)) through a communication network in accordance with communication standards such as 3G, 4G, and / or 5G. Here, the communication module that communicates with a server (not shown) located at a distance may be referred to as a 'distance communication module'.

[0262] The processor (1420) controls the overall operation of the server (1400). For example, the processor (1420) can perform required operations by executing at least one instruction and at least one of the programs of the server (1400).

[0263] Additionally, the DB (1430) may include memory (not shown) and may store at least one of at least one instruction, program, and data required for the server (1400) to perform a predetermined operation within the memory (not shown). Additionally, the DB (1430) may store data required for the server (1400) to perform operations according to a neural network.

[0264] Specifically, in the disclosed embodiment, the server (1400) may store the neural network (1300) described in FIG. 13. The neural network (1300) may be stored in at least one of the processor (1420) and DB (1430). The neural network (1300) included in the server (1400) may be a neural network that has completed training.

[0265] Additionally, the server (1400) can transmit the trained neural network to the communication interface (160) of the electronic device (100) through the communication interface (1410). Then, the electronic device (100) can acquire and store the trained neural network and acquire the desired output data through the neural network.

[0266] According to one embodiment of the present disclosure, an electronic device (100) mounted in a vehicle (1000) may be provided. The electronic device according to one embodiment may include a plurality of cameras. The electronic device according to one embodiment may include a memory for storing at least one instruction. The electronic device according to one embodiment may include at least one processor including a circuit device. By executing at least one instruction individually or collectively by the at least one processor, the electronic device may acquire a plurality of in-vehicle images through the plurality of cameras. By executing at least one instruction individually or collectively by the at least one processor, the electronic device may acquire a set of biometric information corresponding to each of the plurality of in-vehicle images and representing the biometric information of the vehicle driver from each of the plurality of in-vehicle images using a plurality of neural network models. By executing at least one instruction individually or collectively by at least one processor, the electronic device can obtain a set of prompts including at least one prompt that represents the driver's biometric state as text by combining at least one biometric information included in a set of biometric information using a prompt generation model. By executing at least one instruction individually or collectively by said at least one processor, the electronic device for a vehicle can obtain the driving state of the vehicle driver corresponding to the set of prompts using a generative artificial intelligence model.By executing at least one instruction individually or collectively by at least one processor, the electronic device for a vehicle can identify whether the driver of the vehicle is driving normally based on the driving state of the driver corresponding to each of a plurality of vehicle interior images. By executing at least one instruction individually or collectively by at least one processor, the electronic device for a vehicle can output a warning based on the identification that the driver of the vehicle is not driving normally.

[0267] According to one embodiment of the present disclosure, by executing at least one instruction individually or collectively by at least one processor, the electronic device can acquire a first set of biometric information of a vehicle driver based on a first interior image of a vehicle acquired by a first camera among a plurality of cameras using a plurality of neural network models. By executing at least one instruction individually or collectively by at least one processor, the electronic device can acquire a second set of biometric information of a vehicle driver based on a second interior image of a vehicle acquired by a second camera among a plurality of cameras using a plurality of neural network models. By executing at least one instruction individually or collectively by at least one processor, the electronic device can acquire a third set of biometric information of a vehicle driver based on a third interior image of a vehicle acquired by a third camera among a plurality of cameras using a plurality of neural network models.

[0268] According to one embodiment of the present disclosure, a set of biometric information may include at least one of heart rate information obtained from a first neural network model among a plurality of neural network models, eye movement information obtained from a second neural network model among a plurality of neural network models, head movement information obtained from a third neural network model among a plurality of neural network models, or stress information obtained from a fourth neural network model among a plurality of neural networks.

[0269] According to one embodiment of the present disclosure, eye movement information may include an eye movement rate representing a change per reference time of the eye position coordinates of the vehicle driver within the vehicle interior image. Head movement information may include a head movement rate representing a change per reference time of the head position coordinates of the vehicle driver within the vehicle interior image. The reference time may be determined according to the frame rate of each of the plurality of cameras.

[0270] According to one embodiment of the present disclosure, by executing at least one instruction individually or collectively by at least one processor, an electronic device can identify a difference between at least one biometric information included in a set of biometric information and biometric information corresponding to a normal operating state. By executing at least one instruction individually or collectively by at least one processor, the electronic device can obtain at least one prompt based on at least one biometric information included in a set of biometric information or the difference using a prompt generation model.

[0271] According to one embodiment of the present disclosure, the driving state of a vehicle driver may include at least one of a normal driving state, a drowsy driving state, or a drunk driving state. According to one embodiment of the present disclosure, by having at least one instruction executed individually or collectively by at least one processor, an electronic device may acquire the driving state of a vehicle driver corresponding to each of a plurality of vehicle interior images by inputting a prompt set corresponding to each of a plurality of vehicle interior images into a generative artificial intelligence model. By having at least one instruction executed individually or collectively by at least one processor, an electronic device may acquire the driving state of a vehicle driver corresponding to each of a plurality of vehicle interior images by inputting each of a plurality of vehicle interior images and a prompt set corresponding to each of a plurality of vehicle interior images into a generative artificial intelligence model.

[0272] According to one embodiment of the present disclosure, by executing at least one instruction individually or collectively by at least one processor, the electronic device can identify that the driver of the vehicle is driving normally, as the driving state of the driver corresponding to each of the plurality of vehicle interior images is all a normal driving state. By executing at least one instruction individually or collectively by at least one processor, the electronic device can identify that the driver of the vehicle is not driving normally, as at least one of the driving states of the driver corresponding to each of the plurality of vehicle interior images is not a normal driving state.

[0273] According to one embodiment of the present disclosure, by executing at least one instruction individually or collectively by at least one processor, the electronic device can obtain the initial heart rate of the vehicle driver from a first neural network model among a plurality of neural network models based on the vehicle's ignition being turned on. By executing at least one instruction individually or collectively by at least one processor, the electronic device can obtain a plurality of vehicle interior images through a plurality of cameras based on the difference between the initial heart rate and the normal heart rate being included in a threshold range.

[0274] According to one embodiment of the present disclosure, the electronic device may include a display. According to one embodiment of the present disclosure, the electronic device may include a speaker. By having at least one instruction executed individually or collectively by at least one processor, the electronic device may output a warning through at least one of the display or speaker based on at least one of the driver being identified as drowsy driving or drunk driving.

[0275] According to one embodiment of the present disclosure, an electronic device may include a communication interface. By having at least one instruction executed individually or collectively by at least one processor, the electronic device may control the communication interface to transmit to a server at least one of a plurality of in-vehicle images or an in-vehicle image obtained through a vehicle black box, based on at least one of the driver being identified as drowsy driving or drunk driving.

[0276] According to one embodiment of the present disclosure, at least one instruction is executed individually or collectively by at least one processor, thereby controlling the vehicle so that the vehicle's lights output a warning light based on the identification that the driver is drowsy or drunk driving.

[0277] According to one embodiment of the present disclosure, at least one instruction is executed individually or collectively by at least one processor to perform at least one of controlling the driving speed of a vehicle to below a threshold speed or activating the autonomous driving function of a vehicle based on the identification that the driver is drowsy or drunk driving.

[0278] According to one aspect of the present disclosure, a method for controlling a vehicle through an electronic device mounted in the vehicle may be provided. A method according to one embodiment may include the operation of acquiring a plurality of in-vehicle images through a plurality of cameras. The method may include the operation of acquiring a set of biometric information corresponding to each of the plurality of in-vehicle images and representing biometric information of the vehicle driver from each of the plurality of in-vehicle images using a plurality of neural network models. A method according to one embodiment of the present disclosure may include the operation of acquiring a set of prompts including at least one prompt representing the driver's biometric state as text by combining at least one biometric information included in the set of biometric information using a prompt generation model. A method according to one embodiment of the present disclosure may include the operation of acquiring the driving state of the vehicle driver corresponding to the set of prompts using a generative artificial intelligence model. The method may include the operation of identifying whether the vehicle driver is driving normally based on the driving state of the driver corresponding to each of the plurality of in-vehicle images. A method according to one embodiment of the present disclosure may include an operation of outputting a warning based on the identification that the driver of a vehicle is not driving normally.

[0279] A method according to one embodiment of the present disclosure may include an operation of acquiring a first set of biometric information of a vehicle driver based on a first interior image of a vehicle acquired by a first camera among a plurality of cameras using a plurality of neural network models. A method according to one embodiment of the present disclosure may include an operation of acquiring a second set of biometric information of a vehicle driver based on a second interior image of a vehicle acquired by a second camera among a plurality of cameras using a plurality of neural network models. A method according to one embodiment of the present disclosure may include an operation of acquiring a third set of biometric information of a vehicle driver based on a third interior image of a vehicle acquired by a third camera among a plurality of cameras using a plurality of neural network models.

[0280] According to one embodiment of the present disclosure, a set of biometric information may include at least one of heart rate information obtained from a first neural network model among a plurality of neural network models, eye movement information obtained from a second neural network model among a plurality of neural network models, head movement information obtained from a third neural network model among a plurality of neural network models, or stress information obtained from a fourth neural network model among a plurality of neural networks.

[0281] According to one embodiment of the present disclosure, eye movement information may include an eye movement rate representing a change per reference time of the eye position coordinates of the vehicle driver within the vehicle interior image. Head movement information may include a head movement rate representing a change per reference time of the head position coordinates of the vehicle driver within the vehicle interior image. The reference time may be determined according to the frame rate of each of the plurality of cameras.

[0282] A method according to one embodiment of the present disclosure may include an operation of identifying a difference between at least one biometric information included in a set of biometric information and biometric information corresponding to a normal operating state. The method may include an operation of obtaining at least one prompt based on at least one biometric information included in the set of biometric information or the difference using a prompt generation model.

[0283] According to one embodiment of the present disclosure, the driving state of a vehicle driver may include at least one of a normal driving state, a drowsy driving state, or a drunk driving state. A method according to one embodiment of the present disclosure may include an operation of acquiring the driving state of a vehicle driver corresponding to each of a plurality of vehicle interior images by inputting a prompt set corresponding to each of a plurality of vehicle interior images into a generative artificial intelligence model. The method may include an operation of acquiring the driving state of a vehicle driver corresponding to each of a plurality of vehicle interior images by inputting each of a plurality of vehicle interior images and a prompt set corresponding to each of a plurality of vehicle interior images into a generative artificial intelligence model.

[0284] A method according to one embodiment of the present disclosure may include an operation of identifying that the driver of a vehicle is driving normally, as the driving state of the driver corresponding to each of a plurality of vehicle interior images is a normal driving state. The method may include an operation of identifying that the driver of a vehicle is not driving normally, as at least one of the driving states of the driver corresponding to each of a plurality of vehicle interior images is not a normal driving state.

[0285] A method according to one embodiment of the present disclosure may include the operation of obtaining the initial heart rate of a vehicle driver from a first neural network model among a plurality of neural network models based on the vehicle's ignition being turned on. The method may include the operation of obtaining a plurality of vehicle interior images through a plurality of cameras based on the difference between the initial heart rate and the normal heart rate being included in a threshold range.

[0286] A method according to one embodiment of the present disclosure may include the operation of outputting a warning through at least one of a display or a speaker based on at least one of identifying that the driver is drowsy driving or drunk driving.

[0287] A method according to one embodiment of the present disclosure may include an operation in which an electronic device controls a communication interface to transmit to a server at least one of a plurality of in-vehicle images or an external vehicle image obtained through a vehicle black box, based on at least one of the driver being identified as drowsy driving or drunk driving.

[0288] A method according to one embodiment of the present disclosure may include controlling a vehicle so that the vehicle's lights output a warning light based on the identification that the driver is drowsy or drunk driving.

[0289] A method according to one embodiment of the present disclosure may include an operation of controlling the vehicle to control the vehicle's operating speed to below a threshold speed or to activate the vehicle's autonomous driving function based on the identification that the driver is drowsy or drunk driving.

[0290] According to one aspect of the present disclosure, a computer-readable recording medium may be provided that records a program for executing any one of the methods of controlling a vehicle through an electronic device described above and below.

[0291] A method of operation of an electronic device according to one embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the present invention, or may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0292] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory storage medium' simply means that it is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily. For example, a 'non-transitory storage medium' may include a buffer in which data is stored temporarily.

[0293] In addition, the electronic device mounted in a vehicle and the method of operating the electronic device mounted in a vehicle according to the disclosed embodiments may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a product.

[0294] A computer program product may include a software program and a computer-readable storage medium on which the software program is stored. For example, a computer program product may include a product in the form of a software program (e.g., a downloadable app) that is electronically distributed through a manufacturer of an electronic device or an electronic market (e.g., Google Play Store, App Store). For electronic distribution, at least a portion of the software program may be stored on a storage medium or temporarily created. In this case, the storage medium may be a server of the manufacturer, a server of the electronic market, or a storage medium of a relay server that temporarily stores the software program.

[0295] A computer program product may include a storage medium of a server or a storage medium of a client device in a system composed of a server and a client device. Alternatively, if a third device (e.g., a smartphone) is communicationally connected to the server or client device, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include the S / W program itself, which is transmitted from the server to the client device or the third device, or from the third device to the client device.

[0296] In this case, one of the server, the client device, and the third device may execute the computer program product to perform the method according to the disclosed embodiments. Alternatively, two or more of the server, the client device, and the third device may execute the computer program product to perform the method according to the disclosed embodiments in a distributed manner.

[0297] For example, a server (e.g., a cloud server or an artificial intelligence server, etc.) can execute a computer program product stored on the server to control a client device connected to the server in communication to perform a method according to the disclosed embodiments.

[0298] Although the embodiments have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention as defined in the following claims also fall within the scope of the present invention.

Claims

1. In an electronic device (100) mounted in a vehicle (1000), Multiple cameras (110); Memory (130) for storing at least one instruction; and It includes at least one processor (120) including a circuit device, and By having the above at least one instruction executed individually or collectively by the above at least one processor (120), the electronic device (100) A plurality of in-vehicle images are obtained through the plurality of cameras (110) above, and Using multiple neural network models, a set of biometric information corresponding to each of the multiple vehicle interior images and representing the biometric information of the driver of the vehicle is obtained from each of the multiple vehicle interior images. Using a prompt generation model, a set of prompts is obtained that includes at least one prompt representing the driver's biometric state as text by combining at least one biometric information included in the set of biometric information. Using a generative artificial intelligence model, the driving state of the driver of the vehicle corresponding to the prompt set is obtained, and Based on the driving state of the driver corresponding to each of the plurality of vehicle interior images, identifying whether the driver of the vehicle is driving normally, and An electronic device (100) that outputs a warning based on the identification that the driver of the above vehicle is not driving normally.

2. In Paragraph 1, By having the above at least one instruction executed individually or collectively by the above at least one processor (120), the electronic device (100) Using the above plurality of neural network models, a first set of biometric information of the driver of the vehicle is obtained based on a first interior image of the vehicle obtained by the first camera (110-1) among the plurality of cameras, and Using the above plurality of neural network models, a second set of biometric information of the driver of the vehicle is obtained based on the second interior image of the vehicle obtained by the second camera (110-2) among the plurality of cameras, and An electronic device (100) that acquires a third set of biometric information of a driver of a vehicle based on a third internal image of the vehicle acquired by a third camera (110-3) among the plurality of cameras using the above-mentioned neural network models.

3. In Paragraph 1 or 2, The above set of biometric information is, An electronic device (100) comprising at least one of heart rate information obtained from a first neural network model among the plurality of neural network models, eye movement information obtained from a second neural network model among the plurality of neural network models, head movement information obtained from a third neural network model among the plurality of neural network models, or stress information obtained from a fourth neural network model among the plurality of neural networks.

4. In Paragraph 3, The above eye movement information includes an eye movement speed representing a change per reference time of the driver's eye position coordinates within the vehicle interior image, and The head movement information includes a head movement speed representing a change per reference time of the head position coordinates of the driver of the vehicle within the vehicle interior image, and The above reference time is determined according to the frame rate of each of the plurality of cameras, in an electronic device (100).

5. In any one of paragraphs 1 through 4, By having the above at least one instruction executed individually or collectively by the above at least one processor (120), the electronic device (100) Identifying the difference between at least one biometric information included in the above biometric information set and biometric information corresponding to a normal operating state, and An electronic device (100) that obtains at least one prompt based on at least one biometric information included in the set of biometric information or the difference using the above prompt generation model.

6. In any one of paragraphs 1 through 5, The driving condition of the driver of the above vehicle is, It includes at least one of a normal driving state, a drowsy driving state, or a drunk driving state, and By having the above at least one instruction executed individually or collectively by the above at least one processor (120), the electronic device (100) By inputting the prompt set corresponding to each of the plurality of vehicle interior images into the generative artificial intelligence model, the driving state of the driver of the vehicle corresponding to each of the plurality of vehicle interior images is obtained, or An electronic device (100) for obtaining the driving state of the driver of the vehicle corresponding to each of the plurality of vehicle interior images by inputting each of the plurality of vehicle interior images and the prompt set corresponding to each of the plurality of vehicle interior images into the generative artificial intelligence model.

7. In any one of paragraphs 1 through 6, By having the above at least one instruction executed individually or collectively by the above at least one processor (120), the electronic device (100) Identifying that the driver of the vehicle is driving normally, based on the fact that the driving state of the driver corresponding to each of the plurality of vehicle interior images is a normal driving state. An electronic device (100) that identifies that the driver of the vehicle is not driving normally, based on the fact that at least one of the driving states of the driver corresponding to each of the plurality of vehicle interior images is not a normal driving state.

8. In any one of paragraphs 1 through 7, The above electronic device (100) is, Display (140); and It further includes a speaker (150), By having the above at least one instruction executed individually or collectively by the above at least one processor (120), the electronic device (100) An electronic device (100) that outputs a warning message through at least one of the display (140) or the speaker (150) based on at least one of the driver being identified as drowsy driving or drunk driving.

9. In any one of paragraphs 1 through 8, The above electronic device (100) is, Communication interface (160); and It further includes a black box (170), An electronic device (100) that controls the communication interface (160) to transmit to a server at least one of the plurality of vehicle interior images or the vehicle exterior images obtained through the vehicle's black box (170), based on at least one of the identification that the driver is drowsy driving or drunk driving.

10. In any one of paragraphs 1 through 9, By having the above at least one instruction executed individually or collectively by the above at least one processor (120), the electronic device (100) An electronic device (100) that controls the vehicle to output a warning light based on at least one of the identification that the driver is drowsy driving or drunk driving.

11. In any one of paragraphs 1 through 10, By having the above at least one instruction executed individually or collectively by the above at least one processor (120), the electronic device (100) An electronic device (100) that controls the vehicle to control the vehicle's operating speed to below a threshold speed or activate the vehicle's autonomous driving function based on the identification that the driver is drowsy or drunk driving.

12. In any one of paragraphs 1 through 11, By having the above at least one instruction executed individually or collectively by the above at least one processor (120), the electronic device (100) Based on the ignition of the vehicle being turned on, the initial heart rate of the driver of the vehicle is obtained from the first neural network model among the plurality of neural network models, and Based on the difference between the initial heart rate and the normal heart rate being included in a threshold range, a plurality of vehicle interior images of the vehicle are obtained through the plurality of cameras (110), and An electronic device (100) that outputs the warning based on the fact that the difference between the initial heart rate and the normal heart rate is not included in the threshold range.

13. A method of operating an electronic device (100) mounted in a vehicle (1000), The operation of acquiring multiple in-vehicle images through multiple cameras; The operation of obtaining a set of biometric information corresponding to each of the plurality of vehicle interior images and representing the biometric information of the driver of the vehicle from each of the plurality of vehicle interior images using a plurality of neural network models; The operation of obtaining a set of prompts including at least one prompt that represents the driver's biometric state as text by combining at least one biometric information included in the set of biometric information using a prompt generation model; An operation to obtain the driving state of the driver of the vehicle corresponding to the prompt set using a generative artificial intelligence model; An operation to identify whether the driver of the vehicle is driving normally based on the driving state of the driver corresponding to each of the plurality of vehicle interior images; and A method comprising the operation of outputting a warning based on the identification that the driver of the above vehicle is not driving normally.

14. In Paragraph 13, The operation of acquiring multiple in-vehicle images through the above multiple cameras is, The operation of acquiring a first set of biometric information of the driver of the vehicle based on a first interior image of the vehicle acquired by a first camera among the plurality of cameras, using the plurality of neural network models; The operation of acquiring a second set of biometric information of the driver of the vehicle based on a second interior image of the vehicle acquired by a second camera among the plurality of cameras using the plurality of neural network models; and A method comprising the operation of acquiring a third set of biometric information of a driver of a vehicle based on a third internal image of the vehicle acquired by a third camera among the plurality of cameras, using the plurality of neural network models.

15. A computer-readable recording medium having a program recorded thereon for performing the method of any one of paragraphs 13 to 14 on a computer.

Images