Electronic device and method for providing event data of electronic device

Abstract

An electronic device according to one embodiment of the present disclosure comprises: a multi-data sensing module; an event data output module; a memory for storing instructions; and at least one processor, wherein, when executed by the at least one processor, the instructions can cause the electronic device to: acquire multi-data from the multi-data sensing module; analyze the multi-data so as to define an event; determine whether event data related to the event exists; integrate the multi-data into the event data and store same when the event data exists; and provide the integrated and stored event data through the event data output module.

Description

Electronic device and method of providing event data of the electronic device

[0001] The embodiments of the present disclosure relate to an electronic device and a method for providing event data through said electronic device. Specifically, they relate to an electronic device and a method for providing event data that integrates relevant data together to classify them into event data through the analysis of multi-data acquired by the electronic device, and provides the event data to a user.

[0002] With the recent advancement of electronic technology, electronic devices can provide information to users in various forms and through various output methods, such as smartphones, wearable devices, tablet PCs (personal computers), smart TVs (televisions), robots, and drones.

[0003] As an example of providing information to a user through various electronic devices, extended reality (XR) technology, which provides a virtual environment to a user, may include virtual reality (VR), which provides a virtual environment within the user's field of vision; augmented reality (AR), which combines virtual information with a real environment; and mixed reality (MR), which combines AR and VR. Such XR technology may be provided by wearable electronic devices that can provide virtual images within the user's field of vision by being worn on the user's body, such as head-mounted devices or glasses-type devices (e.g., AR glasses). The virtual images provided by wearable electronic devices may include various virtual objects and information generated by artificial intelligence embedded in the electronic device.

[0004] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art in relation to the present disclosure.

[0005] An electronic device according to one embodiment of the present disclosure may include a multi-data sensing module.

[0006] An electronic device according to one embodiment of the present disclosure may include an event data output module.

[0007] An electronic device according to one embodiment of the present disclosure may include a memory for storing instructions.

[0008] An electronic device according to one embodiment of the present disclosure may include at least one processor.

[0009] In an electronic device according to one embodiment of the present disclosure, the instructions, when executed by the at least one processor, cause the electronic device to acquire multidata from the multidata sensing module.

[0010] In an electronic device according to one embodiment of the present disclosure, the instructions, when executed by the at least one processor, can cause the electronic device to analyze the multidata and define an event.

[0011] In an electronic device according to one embodiment of the present disclosure, when the instructions are executed by the at least one processor, the electronic device may determine whether there is event data associated with the event.

[0012] In an electronic device according to one embodiment of the present disclosure, when the instructions are executed by the at least one processor, the electronic device may be configured to integrate and store the multi-data into the event data if the event data exists.

[0013] In an electronic device according to one embodiment of the present disclosure, the instructions, when executed by the at least one processor, may cause the electronic device to provide the integrated stored event data through the event data output module.

[0014] A method for providing event data of an electronic device comprising a multi-data sensing module, an event data output module, and a memory according to one embodiment of the present disclosure may include an operation of acquiring multi-data from the multi-data sensing module.

[0015] In one embodiment of the present disclosure, the method for providing event data may include an operation of defining an event by analyzing the multi-data.

[0016] In one embodiment of the present disclosure, the method for providing event data may include an operation of determining whether there is event data related to the event.

[0017] In one embodiment of the present disclosure, the method for providing event data may include, when the event data exists, an operation of integrating and storing the multi-data into the event data.

[0018] In one embodiment of the present disclosure, the event data providing method may include an operation of providing the integrated stored event data through the event data output module.

[0019] A non-transient computer-readable storage medium (or computer program product) that stores one or more programs may be described.

[0020] One or more programs according to one embodiment may include instructions for acquiring multidata from the multidata sensing module when executed by at least one processor of the electronic device.

[0021] One or more programs according to one embodiment may include instructions for analyzing the multidata to define an event.

[0022] One or more programs according to one embodiment may include a command to determine whether there is event data related to the event.

[0023] One or more programs according to one embodiment may include a command to integrate and store the multi-data into the event data when the event data exists.

[0024] One or more programs according to one embodiment may include a command to provide the integrated stored event data through the event data output module.

[0025] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0026] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments of the present disclosure.

[0027] FIG. 2 is a schematic diagram of an exemplary artificial intelligence system according to various embodiments of the present disclosure.

[0028] FIG. 3 is a block diagram of an electronic device according to one embodiment of the present disclosure.

[0029] FIG. 4 is a flowchart illustrating a method for storing multi-data of an electronic device as event data according to one embodiment of the present disclosure.

[0030] FIG. 5 illustrates examples of various electronic devices capable of acquiring multi-data according to one embodiment of the present disclosure.

[0031] FIG. 6 illustrates an example of multidata that can be obtained by an electronic device according to one embodiment of the present disclosure.

[0032] FIG. 7 illustrates various examples of event data stored in an electronic device according to one embodiment of the present disclosure.

[0033] FIG. 8 is a flowchart illustrating a method for storing multi-data of an electronic device as event data according to one embodiment of the present disclosure.

[0034] FIG. 9 is a flowchart illustrating a method for storing multi-data of an electronic device as event data according to one embodiment of the present disclosure.

[0035] FIG. 10 illustrates an example of determining the correlation between multidata or event data of an electronic device according to one embodiment of the present disclosure.

[0036] FIG. 11 illustrates an example of a plurality of event data that are integratedly managed based on the result of determining the correlation between multi-data or event data of an electronic device according to one embodiment of the present disclosure.

[0037] FIG. 12 illustrates an example of an event that can be managed as event data of an electronic device according to one embodiment of the present disclosure.

[0038] FIG. 13 illustrates various examples of event data stored in an electronic device according to one embodiment of the present disclosure.

[0039] FIG. 14 is a flowchart illustrating a method for generating and providing a response to a user query based on integrated stored event data of an electronic device according to one embodiment of the present disclosure.

[0040] FIG. 15 illustrates providing a visual indicator for multi-data and event data on a display of an electronic device according to one embodiment of the present disclosure.

[0041] FIG. 16 illustrates providing a visual indicator for multi-data and event data on a display of an electronic device according to one embodiment of the present disclosure.

[0042] FIG. 17 illustrates an example of a plurality of electronic devices capable of acquiring multiple data for a single event data according to one embodiment of the present disclosure.

[0043] Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings so that those skilled in the art can easily practice them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0044] With the recent advancements in electronic devices and artificial intelligence technology, wearable electronic devices providing AR can recognize objects in the real-world environment viewed by the user and provide related services in response to the recognized objects. For example, a wearable electronic device can acquire and analyze multi-data recognized from the environment currently viewed by the user to determine the user's events of interest. The wearable electronic device can appropriately process and store multi-data related to the user's events of interest to provide various derived information that can enhance user convenience.

[0045] Furthermore, advancements are being made in technology that generates responses to user queries based on various data acquired from electronic devices. For example, when visual data regarding a user's current location is input into an electronic device, the user's current location can be identified in response, and location information can be provided; similarly, if a user inputs a mathematical problem as visual data, the answer to the problem can be derived and provided to the user.

[0046] Specifically, technology is advancing to understand language through large language models (LLMs) and to summarize long sentences or condense content. Electronic devices acquire result data (e.g., answers to inquiries) regarding user inquiries through LLM servers and provide this acquired data to the user. However, since the result data contains vast amounts of information, users may need to examine the data to find the information they need.

[0047] Furthermore, as such AI-based user response generation technology expands, there is a growing need for methods to efficiently process and record vast amounts of data input in real time.

[0048] Conventional electronic devices have provided users with vast amounts of acquired data by storing them in individual files; consequently, there is a need for a method to analyze related events for each acquired data and manage data recognized as related as continuous data.

[0049] Accordingly, the various embodiments disclosed in this document disclose a technology that integrates relevant data through the analysis of multi-data acquired by an electronic device, classifies them into event data, and provides them to a user.

[0050] The technical problems to be solved in this disclosure are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by those skilled in the art from the description below.

[0051] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments.

[0052] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0053] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0054] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0055] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0056] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0057] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0058] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0059] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0060] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0061] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0062] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0063] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0064] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0065] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0066] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0067] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0068] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0069] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) may support a Peak data rate (e.g., 20 Gbps or more) for eMBB realization, loss coverage (e.g., 164 dB or less) for mMTC realization, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for URLLC realization.

[0070] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0071] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0072] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0073] According to one embodiment, commands or data may be transmitted or received between an electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0074] Figure 2 is a schematic diagram of an exemplary artificial intelligence system.

[0075] Referring to FIG. 2, an artificial intelligence system according to one embodiment may include a user interface (240), a database (250), an application and service component (260), an AI framework (220), and a generative AI model (230).

[0076] A user interface (210) may receive a user query. The input may include user input and / or data obtained or generated by an electronic device (e.g., the electronic device (100) described above, the electronic device (300) of FIG. 3, or the electronic device (1300) of FIG. 13). The data may include images, videos, and / or sensor data generated by at least one processor of the electronic device (e.g., at least one processor (110) or the processor (1310) of FIG. 13)) (e.g., illuminance data around the electronic device obtained from a sensor (170) or sensor hub, attitude data (or orientation data) of the electronic device, temperature inside the electronic device (e.g., temperature of the display (140)) or temperature of at least one processor (110), size information of the display area of ​​the display (140), and / or images obtained through an image sensor (150) of the electronic device). For example, the user query may be in the form of natural language, touch data obtained through a touch circuit included in the display (140) (e.g., used to identify input from a finger and / or stylus), an image, audio, and / or video. Additionally, context information may be transmitted along with the user query. The context information may include various side information related to the time when the user query is input into the artificial intelligence system. For example, there may be application information currently being used by the user or location information of the user. As another example, the user query may also be a non-natural language input that does not generate natural language, such as a design request or modification.In addition, a mixed form of the natural language, images, sounds, and context information described above is also possible. Furthermore, the user interface (240) can output results of the artificial intelligence system to the user. The output may include results (or result information) generated or obtained by the artificial intelligence system based on at least part of the input. The output may be in the form of natural language or specific content, and may also be provided in a form such as an action requested by the user. For example, the output may have a format according to the user settings of the electronic device.

[0077] The AI ​​framework (220) can receive a user query and coordinate and control each component necessary to perform the user's intent. The AI ​​framework (220) may include a prompt design component (221), an APIs / Plugins Management component (223), and an output modification component (225).

[0078] User queries or actions entered in the user interface (240) can be transmitted to a prompt design component (221). The prompt design component (221) can be used to generate prompts suitable for input into a large language model (LLM), a large vision model (LVM), or large multimodal models. The prompt design component (221) may be an AI component that uses machine learning algorithms or neural networks to develop better prompts over time. The prompt design component (221) can generate prompts by accessing a database (250) (e.g., a knowledge component) containing user preference data, a prompt library, and prompt examples, and can transmit them to the large language model (LLM), large vision model (LVM), and / or large multimodal model (LMM).

[0079] The application and plugin management component (223) can perform the role of communicating with external information when there is a request for additional information when user input is transmitted as input to the generative AI model (230). The application and plugin management component (223) establishes a channel to communicate with the outside of the artificial intelligence system through an application programming interface (API), thereby enabling access to various data sources. For example, the application and plugin management component (223) can be used to request other components (e.g., application and service components (260)) that perform feedback (or response) according to the prompt. The acquired information can be used to generate a prompt by the prompt design component (221) together with the user input, or can be used as input to the generative AI model (230). Additionally, the application and plugin management component (223) can request the action through the API if the application or service needs to perform an action that ultimately executes the user query rather than an intermediate result. Information obtained from an external source can be transmitted as input to a generative AI model (230) along with user input.

[0080] The output modification component (225) can finely tune (or adjust) (or change) the output output from the generative AI model (230). For example, the output modification component (225) can determine the relevance (e.g., score) between the output (e.g., content) of the generative AI model and the user input. For example, the output modification component (225) can verify whether the content generated through the Large Language Model (LLM), Large Vision Model (LVM), or Large Multimodal Model (LMM) contains the aforementioned relevance, biased information (e.g., selective information), or harmful information (e.g., violent content or profanity). Additionally, the output modification component (225) can determine the extent to which the output matches the desired result and, if additional processing is required, proceed with the process. Additionally, the output modification component (225) can configure and provide to the user a hint to avoid unwanted output.

[0081] A generative AI model (230) generally refers to an artificial intelligence neural network that generates new forms of data based on user input information. A generative AI model (230) may include an image generation model and / or a language generation model. An image generation model may include a generative adversarial network (GAN) and / or a variational autoencoder (VAE). An example of an image generation model is a diffusion-based generative model that uses the structures of a VAE and a transformer. Additionally, a language generation model is a model trained to output the statistically most appropriate output value based on input values, and representative examples include models such as CHAT-GPT 3 and CHAT-GPT 4. Additionally, it may include large multimodal models (LMMs) capable of recognizing various forms of data input, such as text, images, and voice, and generating new data corresponding to them.

[0082] In one embodiment, the AI ​​framework (220) and / or generative AI model (230) may be included within an AI module (e.g., including a processing circuit) within the electronic device. For example, the AI ​​module may be operatively coupled with at least one processor of the electronic device (e.g., at least one processor (110) or processor (1310)). For example, the AI ​​module may be operatively coupled with a sensor hub of the electronic device for one or more sensors within the electronic device.

[0083] FIG. 3 is a block diagram of an electronic device according to one embodiment of the present disclosure.

[0084] Referring to FIG. 3, an electronic device (101) according to one embodiment of the present disclosure may include a multi-data sensing module (200), an event data output module (210), a memory (130) and / or a processor (120).

[0085] According to one embodiment, the multi-data sensing module (200), the event data output module (210), the memory (130) and / or the processor (120) may be electrically or operationally connected.

[0086] According to one embodiment, the multi-data sensing module (200) can receive multi-data from an external environment where the electronic device (101) is currently located.

[0087] The multi-data sensing module (200) may include an input module (150), a camera module (180), and an audio module (170) to acquire data from an external environment where the electronic device (101) is currently located. That is, multi-data can be acquired from the external environment where the electronic device (101) is currently located through the input module (150), the camera module (180), and the audio module (170).

[0088] According to one embodiment, the multi-data sensing module (200) may include an input module (150).

[0089] The input module (150) may include, for example, a display module (160), a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0090] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user).

[0091] The input module (150) can receive user input from the user to start acquiring multi-data. That is, as the user inputs a multi-data sensing request through the input module (150), the electronic device (101) can start an operation to acquire multi-data present in the external environment.

[0092] For example, the user may provide user input to the "start data recording" button in the input module (150) to acquire multi-data currently occurring in the meeting room in time with the start of the meeting, and in response, the electronic device (101) may start various multi-data acquisition operations detected at the time of the meeting for the meeting event.

[0093] The input module (150) can receive multi-data including text and images from the user. In an event environment to be tracked and managed, the user can provide multi-data including various text and images through the input module (150).

[0094] For example, during a meeting, the user can provide voice data using a microphone, which is an example of an input module (150). That is, if voice data regarding the meeting topic (agenda), meeting content, and key issues is obtained from the user through the microphone during the meeting, this voice data can be provided as multimedia data for the meeting event.

[0095] The input module (150) can obtain user queries from the user. If a user has a query regarding an event of current interest, the user can input the query into the input module (150). User queries may include text data or voice data. However, they are not limited thereto.

[0096] For example, the user can input a query regarding the "topic to be discussed in the meeting" currently being conducted or a query regarding the "date for the next meeting" based on the minutes of a concluded meeting through the input module (150).

[0097] According to one embodiment, the multi-data sensing module (200) may include a camera module (180).

[0098] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0099] The camera module (180) can capture an image, video, or video of the external environment where the electronic device (101) is currently located and acquire it as multi-data.

[0100] For example, if an electronic device (101, e.g., laptop, home camera, robot) is located in a meeting room during a meeting, the camera module (180) within the electronic device (101) can capture images or videos of the faces of meeting participants present in the meeting room and meeting materials displayed on another electronic device. Additionally, the entire progress of the meeting can be captured as video or video. The image, video, and video data acquired by the camera module (180) can be provided as multimedia data regarding the meeting event.

[0101] According to one embodiment, the multidata sensing module (200) may include an audio module (170).

[0102] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102, 104)) (e.g., speaker or headphones) that is directly or wirelessly connected to the electronic device (101).

[0103] The audio module (170) can acquire audio data of the user's voice and the external environment where the current electronic device (101) is located as multi-data. That is, the audio module (170) can receive voice commands or questions from the user. It can also acquire environmental sounds around the user.

[0104] For example, if an electronic device (101) is located in a meeting room during a meeting, the audio module (170) within the electronic device (101) can collect all audio data detected during the meeting. That is, to determine important agenda items and important decisions discussed during the meeting, the audio data of the meeting participants can be detected and used as multi-data for the meeting event.

[0105] According to one embodiment, the event data output module (210) can output event data including multi-data or cumulative data for an event that a user wants to track and manage, which is stored in the electronic device (101).

[0106] The event data output module (210) may include a display module (160) and a sound output module (155) for outputting event data. That is, event data stored in the electronic device (101) can be output from the display module (160) and the sound output module (155). Specifically, when a user requests the output of event data, the event data can be provided in a visual form or an auditory form.

[0107] According to one embodiment, the event data output module (210) may include a display module (160).

[0108] According to various embodiments, the display module (160) can display an initial screen, a settings screen, and / or an execution screen of various applications of the electronic device (101). The display module (160) can provide a user interface (UI) capable of providing various services to the user of the electronic device (101).

[0109] According to various embodiments, the display module (160) may include at least one of a liquid crystal display (LCD), organic light emitting diodes (OLED), and active matrix organic light emitting diodes (AMOLED). The touch circuit may include a touch sensor or a pressure sensor such as a capacitive overlay, a resistive overlay, or an infrared beam.

[0110] According to various embodiments, the display module (160) can perform a display function. To perform the display function, the display module (160) may include a touch circuit. The display module (160) can visually provide the user with the menu, input data, function setting information, and various information of the multi-electronic device (101).

[0111] According to one embodiment, the display module (160) can output event data in various forms in response to a user's request to output event data. That is, when a user requests to output event data, the event data can be provided in a brief form (summary information form), a cumulative management form, or an update form.

[0112] For example, during a meeting on the first topic, multi-data is acquired and stored as event data for the meeting event, and a meeting minutes can be created. If the meeting event existed in the past and event data (e.g., Meeting Minutes Version 1) exists, the acquired multi-data is integrated and stored, and the meeting minutes can be updated (e.g., Meeting Minutes Version 2). At this time, the display module (160) can provide the user with the existing meeting minutes (e.g., Meeting Minutes Version 1) and the updated meeting minutes (e.g., Meeting Minutes Version 2).

[0113] According to one embodiment, the display module (160) can output a response generated in response to a user's query. That is, when a user queries for a specific event, it can provide a response generated based on event data for the specific event.

[0114] For example, if a user asks a question about a specific meeting, a response generated based on event data containing data or accumulated data about the meeting within the electronic device (101) can be provided through the display module (160).

[0115] According to one embodiment, the event data output module (210) may include an audio output module (155).

[0116] According to various embodiments, the acoustic output module (155) can output an acoustic signal to the outside of the electronic device (101). The acoustic output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls.

[0117] According to one embodiment, the sound output module (155) can output event data in the form of voice in response to a user's request for event data output. That is, when a user requests event data output, the event data stored in various forms, such as a brief form (summary information form), a cumulative management form, and an update form, can be converted into voice form and provided through the sound output module (155).

[0118] For example, during a meeting on a specific topic, multi-data is acquired and stored as event data for the meeting event, and a meeting minutes can be created. If the meeting event existed in the past and event data (e.g., Meeting Minutes Version 1) exists, the acquired multi-data is integrated and stored, and the meeting minutes can be updated (e.g., Meeting Minutes Version 2). At this time, the sound output module (155) can provide voice data to the user by reading the existing meeting minutes (e.g., Meeting Minutes Version 1) and the updated meeting minutes (e.g., Meeting Minutes Version 2).

[0119] According to one embodiment, the sound output module (155) can output a response generated according to a user's query. That is, in response to a user's query regarding a specific event, a response generated based on event data can be provided in the form of voice.

[0120] For example, if a user asks a question about a specific meeting, the electronic device (101) can generate a response based on event data containing data or accumulated data about the meeting within the electronic device (101) and provide the generated response to the user in the form of voice.

[0121] According to one embodiment, the memory (130) performs the function of storing a program (e.g., the program (140) of FIG. 1) for processing and controlling the processor (120) of the electronic device (101), an operating system (OS) (e.g., the operating system (142) of FIG. 1), various applications, and / or input / output data, and can store a program that controls the overall operation of the electronic device (101). The memory (130) can store various configuration information required for processing functions related to various embodiments of the present disclosure in the electronic device (101). The memory (130) can store executable instructions. For example, the memory (130) can store instructions that cause the electronic device (101) to perform operations when executed by the processor (120). For example, the instructions may be stored in a computer-readable storage medium. The storage medium may be tangible and non-transitory. The memory (130) and / or storage medium may store one or more programs containing instructions.

[0122] In one embodiment, the memory (130) can store multi-data obtained from the multi-data sensing module (200).

[0123] In one embodiment, the multidata may include at least one image, video, video, text, and voice data acquired by the multidata sensing module (200). In one embodiment, the multidata may include at least one image, video, video, text, and voice data acquired by the input module (150), camera module (180), and audio module (170).

[0124] For example, when a meeting regarding the first topic is being held, the multidata may include the faces of the meeting participants, the number of meeting participants, the meeting location, the meeting time, the content of the meeting participants' speech, and at least one image, video, video, text, and voice data related to the meeting content, and this may be stored in memory (130).

[0125] For example, when a user is assembling Lego while wearing or using an electronic device (101), the type of Lego currently being assembled, the assembly progress status, and the assembly duration obtained by the multi-data sensing module (200) may be included in the multi-data, and this may be stored in the memory (130).

[0126] In one embodiment, the memory (130) can store various information for defining events.

[0127] The memory (130) can store information, methods, or criteria for determining related events from multi-data obtained from the multi-data sensing module (200) or from a query received from a user.

[0128] An event can refer to a specific situation related to key information and data that the user intends to focus on tracking and managing.

[0129] A method for determining events related to multidata or user queries may include considering the usage history of the electronic device (101) and / or the usage history of applications existing within the electronic device (101), information stored in applications, and the location and time where the multidata was acquired. For example, the usage history of the electronic device (101) may include applications and functions previously or last used by the user, and tasks that have not yet been terminated. Additionally, the application usage history may include user-specific information contained in applications, search history within applications, and whether functions within applications have been terminated.

[0130] A method for determining an event related to multi-data or a user query may include analyzing at least one image, video, or text obtained from a multi-data sensing module (200) and defining it as an event if it corresponds to key information and data that the user intends to focus on tracking and managing.

[0131] The memory (130) can store events defined from multi-data obtained from the multi-data sensing module (200) or events defined from a query received from a user.

[0132] For example, while a meeting regarding the first topic is in progress, the electronic device (101) may acquire multi-data including the location where the meeting is taking place, the faces of the people attending the meeting, the schedule within the user application, and the meeting materials within the monitor. Accordingly, by analyzing the acquired multi-data, the electronic device (101) may determine that the meeting currently in progress corresponds to the "weekly meeting regarding the first topic held every Thursday." Based on the analyzed multi-data, the electronic device (101) may define the "meeting regarding the first topic" as an event. At this time, the memory (130) may store the "meeting regarding the first topic" event, the criteria for judgment, judgment information, and the judgment process.

[0133] For example, when a user is assembling Lego while wearing or using an electronic device (101), the electronic device (101) may acquire multi-data including the type of Lego currently being assembled, the status of the assembly, the time at which the assembly is performed, and the duration of the assembly. The electronic device (101) may analyze the acquired multi-data to determine that the user is currently assembling Lego as a hobby activity. Based on the results of the multi-data analysis, the electronic device (101) may define "Lego assembly hobby activity" as an event. At this time, the memory (130) may store the "Lego assembly hobby activity" event, the criteria for judgment, judgment information, and the judgment process.

[0134] In one embodiment, the memory (130) can store event data for events that the user wants to track and manage.

[0135] Event data may refer to integrated data that accumulates and stores data regarding a specific event defined from multi-data including text, image, and video data. Event data may refer to data in the form where multiple types of multi-data regarding a specific event are stored in an integrated manner.

[0136] For example, when a meeting regarding the first topic is being held, the multi-data may include the faces of the meeting participants, the number of meeting participants, the meeting location, the meeting time, the content of the meeting participants' speech, and at least one image, video, video, text, and voice data related to the meeting content. At this time, the electronic device (101) may analyze the acquired multi-data to define the "meeting regarding the first topic" as an event, and the acquired multi-data may be stored as event data for the "meeting regarding the first topic" (e.g., meeting minutes version 1).

[0137] On the other hand, if event data for "meeting on the first topic" (e.g., meeting minutes version 1) has already been stored in the memory of the electronic device (101), multi-data acquired in the stored event data (e.g., meeting minutes version 1) may be integrated and accumulated, or the stored event data may be updated and stored according to the included data (e.g., meeting minutes version 2).

[0138] For example, when a user assembles Lego while wearing or using an electronic device (101), the electronic device (101) can acquire multi-data including the type of Lego currently being assembled, the status of the assembly, the time at which the assembly is being performed, and the duration of the assembly. At this time, the electronic device (101) can analyze the multi-data to define the "Lego assembly hobby activity" as an event, and can store the acquired multi-data in memory (130) as event data for the "Lego assembly hobby activity" (e.g., Day 1 of Lego assembly).

[0139] On the other hand, if event data for "Lego assembly hobby activity" (e.g., Lego assembly day n) has already been stored in memory within the electronic device (101), multi-data may be integrated into the stored event data (e.g., Lego assembly day n) and accumulated, or the stored event data may be updated and stored according to the included data (e.g., Lego assembly day n + m).

[0140] In one embodiment, the memory (130) can store information about a plurality of electronic devices that intend to acquire multi-data.

[0141] The electronic device (101) may include a smartphone, a wearable device (e.g., a head-mounted device or a glasses-type device), a tablet PC, a smart TV, a robot, or a drone. The memory (130) may store information about multiple electronic devices (101) configured to acquire multi-data for a specific event. Additionally, the memory (130) may store information regarding whether user authorization has been granted to the electronic device (101) that intends to acquire multi-data for a specific event.

[0142] For example, if a meeting regarding the first topic is in progress and a first computer, a second computer, a first laptop, a first camera, and a second camera are currently present in the meeting room, the electronic device (101) for acquiring multi-data regarding the first topic as the meeting progresses may be limited to the first computer, the first laptop, and the second camera. Additionally, the first computer, the first laptop, and the second camera may be electronic devices (101) authorized by the user to acquire multi-data.

[0143] According to one embodiment, the processor (120) may be electrically or operationally connected to a multi-data sensing module (200), an event data output module (210), and a memory (130). The processor (120) may control the functions and operations of the multi-data sensing module (200), the event data output module (210), and the memory (130). The processor (120) may include at least one processing circuitry, and the processor (120) may include at least one processor.

[0144] The operations of the electronic device (101) or processor (120) described in FIGS. 1 through 17 may be performed individually or collectively by at least one processor included in the processor (120). The operations of the processor (120) illustrated in FIGS. 1 through 17 may be performed according to the execution of instructions contained in memory (130). For example, in this document, an embodiment in which the electronic device (101) can perform any operation may be interpreted to mean that it is performed by at least one instruction stored in the processor (120) and / or memory (130).

[0145] According to various embodiments, the processor (120) may perform the function of controlling the overall operation of the electronic device (101) and the flow of signals between internal components, and processing data. The processor (120) may include, for example, a central processing unit (CPU), an application processor, and / or a communication processor. The processor (120) may include a single-core processor or a multi-core processor. The processor (120) may be composed of at least one processor. For example, the processor (120) may include an application processor (e.g., the main processor (121) of FIG. 1) and a sensor hub processor (e.g., the auxiliary processor (123) of FIG. 1). For example, the sensor hub processor is operatively connected to a multi-data sensing module (200) and can transmit multi-data information detected and / or sensed through the multi-data sensing module (200) to an application processor.

[0146] According to one embodiment, the processor (120) can acquire multidata from a multidata sensing module (200). The processor (120) can acquire multidata from a multidata sensing module (200) present in a plurality of electronic devices, and the plurality of electronic devices may include a plurality of electronic devices authorized by a user to acquire multidata for a specific event.

[0147] According to one embodiment, the processor (120) can define an event related to acquired multi-data and search for whether the event data related thereto exists in the electronic device (101). An event may refer to a specific situation related to key information and data that the user intends to focus on tracking and managing. The processor (120) can define an event based on image, video, text, or audio data acquired from the multi-data sensing module (200) and search for whether the event data related thereto exists in the electronic device (101).

[0148] According to one embodiment, if there is event data related to an event defined from multi-data, the processor (120) may integrate and store the multi-data into the event data and provide the stored event data to the user through the event data output module (210). Specifically, the processor (120) may not simply accumulate and store the acquired multi-data in the existing event data, but may update the previously stored event data to reveal changes and additions to the data by integrating and storing the acquired multi-data, and store it in memory (130).

[0149] According to one embodiment, the processor (120) can provide a visual indicator of multi-data being integrated into event data through a display module (160) when multi-data is integrated into event data and stored.

[0150] According to one embodiment, if event data related to an event defined from multidata does not exist in the electronic device (101), the processor (120) may newly store the multidata as event data for the defined event and provide the newly stored event data to the user through the event data output module (210).

[0151] According to one embodiment, the processor (120) can provide a visual indicator through the display module (160) that the multi-data is newly stored as event data for a defined event when the multi-data is newly stored as event data for a defined event.

[0152] According to one embodiment, the processor (120) receives a query from a user and can search for whether event data regarding an event recognized as relevant to the query exists in the electronic device (101). That is, it can search for whether event data related to the user query is stored in the electronic device (101).

[0153] According to one embodiment, if integrated event data related to an event defined from a user query exists in the electronic device (101), the processor (120) can generate a response to the query based on the integrated event data stored in the electronic device (101) and provide it to the user.

[0154] According to one embodiment, if the processor (120) does not have integrated stored event data related to an event defined from a user query in the electronic device (101), the electronic device (101) can generate a response to the query based on data obtained from the multi-data sensing module (200) in real time and provide it to the user.

[0155] According to various embodiments, an operation performed in the electronic device (101) may be performed and / or executed by at least one instruction stored in the processor (120) and / or memory (130). For example, in this document, an embodiment in which the electronic device (101) can perform any operation may be interpreted to mean that it is performed by at least one instruction stored in the processor (120) and / or memory (130).

[0156] FIG. 4 is a flowchart illustrating a method for storing multidata of an electronic device as event data according to one embodiment of the present disclosure. (See FIGS. 5 to 7)

[0157] In the following embodiments, each operation of FIG. 4 may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation of FIG. 4 may be changed, and at least two operations may be performed in parallel.

[0158] According to one embodiment, the 1000 to 1400 operations of FIG. 4 can be understood as being performed in a processor (e.g., processor (120) of FIG. 3) of an electronic device (e.g., electronic device (101) of FIG. 3).

[0159] In various embodiments, before performing the operation 1000, the electronic device (101) may perform an operation to establish communication (e.g., pairing) with an external electronic device (e.g., electronic device (102, 104) of FIG. 1).

[0160] In one embodiment, the processor (120) of the electronic device (101) (or the electronic device (101)) can acquire multidata from the multidata sensing module (200) (1000). The processor (120) can acquire multidata including various images, audio, video, and text from the external environment where the electronic device (101) is currently located through the multidata sensing module (200).

[0161] For example, multi-data obtained from an external environment may include the voice of the user of the electronic device (101), conversations with people around them, lecture content, the faces of meeting participants, and meeting materials displayed on the monitor screen.

[0162] For example, when a user is assembling Lego while wearing or using an electronic device (101), the multi-data acquired from an external environment may include the type of Lego currently being assembled, the assembly progress status, and the assembly duration acquired by the multi-data sensing module (200), and this may be stored in memory (130).

[0163] The processor (120) can define events (1100) by analyzing the acquired multi-data. An event may refer to a situation related to key information and data that the user intends to focus on tracking and managing. Criteria for determining events related to multi-data may include the usage history of the electronic device (101) and / or the usage history of applications existing within the electronic device (101), information stored in the applications, and the location and time where the multi-data was acquired, and events included in the acquired multi-data can be defined based on the criteria.

[0164] For example, when a meeting regarding the first topic is in progress, the processor (120) can acquire and analyze the location where the meeting takes place, the faces of the people attending the meeting, the schedule within the user application, and the meeting materials within the monitor as multi-data. Based on the multi-data analysis, the processor (120) can determine that the currently ongoing meeting corresponds to the "weekly meeting regarding the first topic held every Thursday." That is, the currently ongoing meeting is related to the "meeting regarding the first topic," and the "meeting regarding the first topic" can be defined as an event.

[0165] For example, when a user assembles Lego while wearing or using an electronic device (101), the electronic device (101) may acquire multi-data such as the type of Lego currently being assembled, the status of assembly progress, the time at which assembly is performed, and the duration of assembly. The electronic device (101) may analyze the acquired multi-data to determine that the user is currently engaging in Lego assembly as a hobby activity. Based on the results of the multi-data analysis, the electronic device (101) may define the “Lego assembly hobby activity” as an event.

[0166] If event data related to a defined event exists in the electronic device (101), the processor (120) can integrate the acquired multi-data with the existing event data and store it in memory (130) (1200, 1300). Event data may refer to integrated data that is recognized as being related to the event defined from the multi-data. That is, event data may refer to data in the form of multiple multi-data for a specific event that are integrated and stored. A specific method for determining the relationship between multi-data, the event, and event data is described later in FIG. 10.

[0167] For example, when a meeting regarding the first topic is being held, the face of the corresponding meeting participant, the number of meeting participants, the meeting location, the meeting time, the content of the meeting participant's speech, and at least one image, video, video, text, and voice data related to the meeting content may constitute multi-data. Since the acquired multi-data is recognized as being related to the "meeting regarding the first topic" corresponding to the defined event, it may be integrated into and accumulated or updated and stored in the event data regarding the "meeting regarding the first topic" stored in the electronic device (101). The integrated, accumulated, or updated event data may be stored in memory (130) and provided upon user request.

[0168] For example, when a user assembles Lego while wearing or using an electronic device (101), the electronic device (101) can acquire multi-data including the type of Lego currently being assembled, the status of the assembly, the time at which the assembly is being performed, and the duration of the assembly. At this time, the electronic device (101) can analyze the multi-data to define the "Lego assembly hobby activity" as an event, and can store the acquired multi-data in memory (130) as event data for the "Lego assembly hobby activity" (e.g., Day 1 of Lego assembly).

[0169] On the other hand, if event data for "Lego assembly hobby activity" (e.g., Lego assembly day n) has already been stored in memory within the electronic device (101), multi-data may be integrated into the event data (e.g., Lego assembly day n) stored in memory (130) and accumulated, or the stored event data may be updated and stored according to the included data (e.g., Lego assembly day n + m).

[0170] The processor (120) can provide the integrated stored event data to the user through the event data output module (210) (1400). The processor (120) can analyze common data information between multi-data and event data and summarize the integrated stored event data based on the analyzed data information. The processor (120) can provide the event data, in which multi-data is integrated and stored, to the user in a brief form (summary information form), a cumulative management form, or an update form through the event data output module (210). However, it is not limited thereto.

[0171] For example, the processor (120) may acquire multi-data during a meeting on a first topic and store it as event data for the first meeting. If a first meeting event existed in the past and event data for it (e.g., meeting minutes version 1) exists, the processor (120) may integrate and store the acquired multi-data to generate an updated meeting minutes (e.g., meeting minutes version 2). At this time, the processor (120) may store the existing meeting minutes (e.g., meeting minutes version 1) and the updated meeting minutes (e.g., meeting minutes version 2) in memory (130) and provide them upon user request.

[0172] For example, the processor (120) can analyze common data information between multi-data acquired during the first topic meeting and event data stored in the past based on the first meeting event (e.g., meeting minutes version 1), summarize the integrated stored event data (e.g., meeting minutes version 2) based on the analyzed data information, and store it in memory (130). Subsequently, based on a user's request, the integrated stored event data (e.g., meeting minutes version 2) and a summary of the integrated stored event data (e.g., meeting minutes version 2) can be provided to the user.

[0173] For example, when a user is assembling Lego while wearing or using the electronic device (101), the electronic device (101) can analyze common data information between acquired multi-data and event data stored based on past Lego assembly hobby activity events, based on the completion of the assembly of the Lego currently being assembled. Based on the analyzed data information, the electronic device (101) can summarize the event data stored in the memory (130) (e.g., Lego assembly day n, Lego assembly day n + m) and provide it to the user.

[0174] FIG. 5 illustrates examples of various electronic devices capable of acquiring multi-data according to one embodiment of the present disclosure.

[0175] Referring to FIG. 5, an example of an electronic device capable of acquiring various multi-data through a multi-data sensing module (200) according to one embodiment can be seen.

[0176] The processor (120) (or electronic device (101)) of the electronic device (101) can obtain multidata from the multidata sensing module (200).

[0177] The electronic device (101) capable of acquiring multi-data may include various electronic devices such as smartphones, wearable devices (e.g., head-mounted devices or glasses-type devices), tablet PCs, smart TVs, robots, drones, cameras (home cameras), and microphones.

[0178] 5-A relates to a glasses-type wearable electronic device with a built-in multi-data sensing module (200). For example, when a user moves from place to place while wearing the glasses-type wearable electronic device (101-1), the electronic device (101) can acquire multi-data including the location, time, and path of the place the user is moving to through the built-in multi-data sensing module (200). Additionally, when the user is currently solving a math problem, the electronic device (101) can acquire image data of the math problem the user is looking at through the built-in multi-data sensing module (200).

[0179] 5-B relates to a smartphone (101-2) or camera with a built-in multi-data sensing module (200). For example, when a user takes a picture of an object of interest using a smartphone or camera, the electronic device (101) can acquire image and video data of the object of interest through the built-in multi-data sensing module (200).

[0180] FIG. 6 illustrates an example of multidata that can be obtained by an electronic device according to one embodiment of the present disclosure.

[0181] Referring to FIG. 6, examples of multidata that can be obtained from a multidata sensing module (200) by an electronic device (101) in an environment where a specific event is taking place can be seen.

[0182] For example, a meeting on a specific topic is currently in progress in the conference room in real time. Multiple electronic devices (101) present in the conference room can acquire various multi-data generated during the meeting through a multi-data sensing module (200). The multi-data for the meeting may include at least one image, video, video, text, and voice data related to the faces of the meeting participants (6-A, 6-B, 6-C, 6-D, 6-E), the number of meeting participants, the meeting location, the meeting time, the meeting content displayed on the screen (6-F), and the speech content of the meeting participants.

[0183] FIG. 7 illustrates various examples of event data stored in an electronic device according to one embodiment of the present disclosure.

[0184] Referring to Fig. 7, examples of various forms of event data for a specific event can be seen.

[0185] If event data related to an event defined from multi-data exists in the electronic device (101), the processor (120) can integrate the acquired multi-data into the existing event data, store it, and provide it to the user. However, if event data related to an event defined from multi-data does not exist in the electronic device (101), the processor (120) can store the acquired multi-data as new event data and provide it to the user.

[0186] Referring to 7-A through 7-C, it can be seen that when multiple meetings are held on the same topic (event), various forms of event data are generated, such as brief forms (summary information forms), cumulative management forms, and update forms for the same topic.

[0187] For example, if multiple meetings for a "patent workshop" are held, the electronic device (101) can acquire and analyze multi-data including meeting attendees, meeting content (e.g., workshop goals, topics (agenda), meeting keywords, action items), and meeting time, and define the "patent workshop meeting" as an event.

[0188] If there is no event data related to the "patent workshop meeting" in the electronic device (101), the electronic device (101) can store the acquired multi-data as new event data in various forms for the "patent workshop meeting." That is, after the first meeting for the "patent workshop meeting," the electronic device (101) can generate event data (first meeting minutes) including the meeting date, meeting time, meeting attendees, meeting topic (agenda), and tasks to be done (action items) (7-A).

[0189] Subsequently, when a second meeting regarding the "patent workshop" is held, since event data (minutes of the first meeting) related to the "patent workshop meeting" exists in the electronic device (101), the electronic device (101) can integrate the multi-data obtained from the second meeting into the existing event data (minutes of the first meeting) regarding the "patent workshop meeting" to store cumulatively and update. That is, minutes of the second meeting can be created in a brief form (summary information form), a cumulative management form, and an update form (7-B, 7-C).

[0190] The electronic device (101) can determine completed items among the action items recorded in the first meeting through multi-data analysis obtained during the second meeting process and update them in the second meeting minutes. That is, it can mark completed items among the action items recorded in the first meeting in the second meeting minutes (7-B-2) and update newly added outputs (7-B-1). For completed items among the action items, the time of completion can also be recorded along with the completion mark.

[0191] Additionally, the electronic device (101) can create a second meeting minutes by checking the file for the received "Patent Workshop" (Workshop deliverables in FIG. 7-C) through checking the email history when emails are exchanged between meeting participants during the second meeting, and by determining information that no longer needs to be checked and adding a brief summary (7-C-1).

[0192] Finally, when the processor (120) determines that the meeting for the "patent workshop" has concluded, it can generate and save a brief-type meeting minutes that allows verification of the overall meeting process, content, and outputs. That is, when a user makes a user inquiry regarding the total duration of the "patent workshop" meeting, the electronic device (101) can generate a user response based on the brief-type meeting minutes and provide the user response through the event data output module (210).

[0193] FIG. 8 is a flowchart illustrating a method for storing multi-data of an electronic device as event data according to one embodiment of the present disclosure.

[0194] In the following embodiments, each operation of FIG. 8 may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation of FIG. 8 may be changed, and at least two operations may be performed in parallel.

[0195] According to one embodiment, operations 2000 to 2410 of FIG. 8 can be understood as being performed in a processor (e.g., processor (120) of FIG. 3) of an electronic device (e.g., electronic device (101) of FIG. 3).

[0196] Since the operations of 2000 to 2400 (excluding 2310 and 2410) of FIG. 8 according to one embodiment are substantially the same as the operations of 1000 to 1400 of FIG. 4 described above, a detailed description thereof may be replaced by the description of FIG. 4.

[0197] In one embodiment, the processor (120) obtains first multidata (e.g., multidata obtained from a second meeting related to a patent workshop) from a multidata sensing module (200), and analyzes the first multidata to define a first event (e.g., patent workshop) (2000, 2100).

[0198] The processor (120) determines whether there is a first event data (e.g., minutes of the first meeting of the patent workshop) that is related to the first event, and if the first event data exists in the electronic device (101) (e.g., 2200), the first multi-data can be integrated into the first event data and stored (2300). That is, by analyzing the multi-data obtained from the second meeting related to the patent workshop, the "patent workshop" is defined as an event, and if the minutes of the first meeting regarding the "patent workshop" stored in the electronic device (101) exist, the multi-data obtained from the second meeting can be integrated into the minutes of the first meeting to create the minutes of the second meeting.

[0199] The processor (120) can provide integrated stored event data through the event data output module (210) (2400). That is, the processor (120) can provide a second meeting minutes in which multi-data obtained from the second meeting is integrated stored through the event data output module (210).

[0200] The processor (120) determines whether there is a first event data (e.g., minutes of the first patent workshop) related to the first event, and if the first event data does not exist in the electronic device (101) (No in 2200), the first multi-data can be stored in memory (130) as new first event data (2310). That is, by analyzing the multi-data obtained from the second meeting related to the patent workshop, the "patent workshop" is defined as an event, and if the minutes of the first meeting regarding the "patent workshop" stored in the electronic device (101) do not exist, the multi-data obtained from the second meeting can be stored as new minutes.

[0201] The processor (120) can provide the first event data through the event data output module (210) (2410). That is, the processor (120) can provide the meeting minutes in which the multi-data obtained from the second meeting is newly stored through the event data output module (210).

[0202] FIG. 9 is a flowchart illustrating a method for storing multi-data of an electronic device as event data according to one embodiment of the present disclosure.

[0203] In the following embodiments, each operation of FIG. 9 may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation of FIG. 9 may be changed, and at least two operations may be performed in parallel.

[0204] According to one embodiment, the 3000 to 3600 operations of FIG. 9 can be understood as being performed in a processor (e.g., processor (120) of FIG. 3) of an electronic device (e.g., electronic device (101) of FIG. 3).

[0205] Since the 3000 to 3300 operations of FIG. 9 according to one embodiment are substantially the same as the 1000 to 1300 operations of FIG. 4 described above, a detailed description thereof may be replaced by the description of FIG. 4.

[0206] In one embodiment, the processor (120) obtains first multidata (e.g., multidata obtained from a second meeting related to a patent workshop) from a multidata sensing module (200), and analyzes the first multidata to define a first event (e.g., a patent workshop) (3000, 3100).

[0207] The processor (120) determines whether there is a first event data (e.g., minutes of the first patent workshop) related to the first event, and if the first event data exists in the electronic device (101), it can generate a first-1 event data (e.g., minutes of the second patent workshop) by integrating and storing the first multi-data into the first event data (3300). That is, by analyzing the multi-data obtained from the second meeting related to the patent workshop, the "patent workshop" is defined as an event, and if the minutes of the first patent workshop regarding the "patent workshop" stored in the electronic device (101) exist, the minutes of the second patent workshop can be generated by integrating and storing the multi-data obtained from the second patent workshop meeting into the minutes of the first patent workshop.

[0208] The processor (120) can re-acquire multi-data from the multi-data sensing module (200) (3400).

[0209] The processor (120) determines whether the reacquired multi-data is related to the first event, and if the reacquired multi-data is recognized as being related to the first event (example of 3500), the reacquired multi-data can be integrated into the first-1 event data and stored (3600).

[0210] In other words, if the reacquired multi-data is the 3rd Patent Workshop meeting relevant to the "Patent Workshop," the multi-data acquired from the 3rd Patent Workshop meeting can be integrated and stored in the 2nd Patent Workshop meeting minutes. (Creation of 3rd Patent Workshop Meeting Minutes)

[0211] The processor (120) determines whether the reacquired multidata is related to the first event, and if the reacquired multidata is not related to the first event (No in 3500), it can analyze the reacquired multidata to define a second event (e.g., trademark workshop) (3510).

[0212] If second event data (e.g., minutes of a primary trademark workshop) related to the second event exists in the electronic device (101) (3520), the processor (120) can integrate the reacquired multi-data into the second event data and store it (3530).

[0213] That is, if the reacquired multi-data is not related to the "patent workshop," the reacquired multi-data can be analyzed and defined as a "trademark workshop" event. If the processor (120) has a primary trademark workshop meeting minutes related to the "trademark workshop" in the electronic device (101), the processor (120) can integrate and store the reacquired multi-data in the primary trademark workshop meeting minutes (creation of secondary trademark workshop meeting minutes).

[0214] FIG. 10 illustrates an example of determining the correlation between multidata or event data of an electronic device according to one embodiment of the present disclosure.

[0215] Referring to FIG. 10, examples of methods and criteria for determining the correlation between multi-data and / or event data acquired by an electronic device (101) can be seen. For example, the electronic device (101) may define events related to the multi-data and / or event data, and determine the correlation by calculating a correlation index between the defined events.

[0216] The relevance index can be determined based on key object information constituting multi-data and / or event data, voice data, text data, key keywords or information stored in the electronic device (101), and application usage history.

[0217] Additionally, application data installed within the electronic device (101) can be utilized to determine the relevance between the data of the electronic device (101). For example, the electronic device (101) can determine relevance by comparing the context and details of an event through schedule information stored in a calendar application within the electronic device (101), location information obtained from a location application, and messages (conversation content) obtained from a communication application.

[0218] Referring to FIG. 10, examples of methods and criteria for determining the relationship between multi-data and / or event data based on image data obtained from an electronic device (101) can be seen.

[0219] Meeting 10-A has A, B, C, D, and E as attendees, and the meeting materials contain content regarding K topics (e.g., AI, patent, idea). Meeting 10-B has F, G, H, and I as attendees, and the meeting materials contain content regarding L topics (e.g., UX (user experience), usability testing, new concept). Meeting 10-C has A, B, C, D, and J as attendees, and the meeting materials contain content regarding K' topics (e.g., AI agent, patent, specification).

[0220] When comparing in terms of meeting attendees, it can be seen that the correlation index between meetings 10-A and 10-C is relatively high, as four identical attendees (A, B, C, D) participated in both meetings 10-A and 10-C.

[0221] When comparing the content of the meetings, it can be seen that the relevance index of the 10-A and 10-C meetings is relatively high, as they both include content regarding AI and patents.

[0222] In summary, the multi-data obtained from the 10-A and 10-C meetings corresponds to data regarding common events that are recognized as relevant, so they can be integrated, stored, and tracked as the same event data.

[0223] According to one embodiment, the relationship between multi-data and / or event data can be determined based on voice data obtained from an electronic device (101).

[0224] For example, the electronic device (101) can acquire voice data of a person attending the meeting during the meeting and identify the meeting attendees by analyzing the acquired voice data. By analyzing the voice data acquired during the 10-1 meeting, M, N, and O can be identified as the attendees of the 10-1 meeting. By analyzing the voice data acquired during the 10-2 meeting, P, Q, and R can be identified as the attendees of the 10-2 meeting. Additionally, by analyzing the voice data acquired during the 10-3 meeting, M, N, and S can be identified as the attendees of the 10-3 meeting. At this time, when comparing in terms of meeting attendees, it can be determined that the correlation index between the 10-1 meeting and the 10-3 meeting is relatively high, as it appears that the same two attendees (M and N) participated in both the 10-1 meeting and the 10-3 meeting.

[0225] For example, an electronic device (101) can acquire voice data of a person attending a meeting during the meeting and analyze the acquired voice data to identify the content of the utterance. By analyzing the content of the utterance of the 10-1 meeting, a T topic (e.g., trademark infringement, trademark application, trademark similarity) can be identified as meeting material. By analyzing the content of the utterance of the 10-2 meeting, a U topic (e.g., marketing, sales performance, advertising) can be identified as meeting material. By analyzing the content of the utterance of the 10-3 meeting, a T' topic (e.g., patent infringement, patent application, intellectual property rights) can be identified as meeting material. At this time, when comparing in terms of meeting material, it can be determined that the relevance index is relatively high because the meeting materials of the 10-1 meeting and the 10-3 meeting commonly include meeting content (T, T') related to intellectual property rights.

[0226] FIG. 11 illustrates an example of a plurality of event data that are integratedly managed based on the result of determining the correlation between multi-data or event data of an electronic device according to one embodiment of the present disclosure.

[0227] Referring to Fig. 11, it can be seen that event data is stored separately for each event.

[0228] When a first meeting for the "Topic A" event is held, the electronic device (101) can acquire multi-data from the first meeting of Topic A and define "Topic A" as an event. The electronic device (101) can generate and store the acquired multi-data as event data for Topic A. That is, a meeting minutes (v1.0 (version 1.0)) for Topic A can be generated.

[0229] Subsequently, if a first meeting regarding the "Topic B" event is held, the electronic device (101) can acquire multi-data from the first meeting of Topic B and define "Topic B" as an event. If the electronic device (101) does not recognize a correlation between the previously defined "Topic A" and "Topic B," it can generate and store the acquired multi-data as event data for Topic B. That is, a meeting minutes of Topic B (v1.0 (version 1.0)) can be generated.

[0230] Next, if a second meeting regarding the "Topic A" event is held, the electronic device (101) can acquire multi-data from the second meeting of Topic A and define "Topic A" as an event. At this time, the electronic device (101) can update the multi-data acquired from the second meeting of Topic A by integrating it into the previously stored Topic A meeting minutes (v1.0 (version 1.0)) and storing it, assuming that there is a correlation between the previously stored event data regarding "Topic A," namely the Topic A meeting minutes (v1.0 (version 1.0)) and the multi-data acquired from the second meeting of Topic A. That is, the electronic device (101) can create and store the Topic A meeting minutes (v1.1 (version 1.1)).

[0231] Next, if a first meeting for the "Topic C" event is held, the electronic device (101) can acquire multi-data from the first meeting of Topic C and define "Topic C" as an event. If the electronic device (101) does not recognize a relationship between the previously defined "Topic A and Topic B" and "Topic C," it can generate and store the acquired multi-data as event data for Topic C. That is, a meeting minutes of Topic C (v1.0 (version 1.0)) can be generated.

[0232] Subsequently, if a third meeting regarding the "Topic A" event is held, the electronic device (101) can acquire multi-data from the third meeting of Topic A and define "Topic A" as an event. At this time, the electronic device (101) can update the multi-data acquired from the third meeting of Topic A by integrating it into the previously stored Topic A meeting minutes (v1.1 (version 1.1)) and storing it, assuming that there is a correlation between the previously stored event data regarding "Topic A," namely the Topic A meeting minutes (v1.1 (version 1.1)) and the multi-data acquired from the third meeting of Topic A. That is, the electronic device (101) can create and store the Topic A meeting minutes (v1.2 (version 1.2)).

[0233] Next, if a second meeting for the "Topic C" event is held, the electronic device (101) can acquire multi-data from the second meeting of Topic C and define "Topic C" as an event. If the electronic device (101) does not recognize a relationship between the previously defined "Topic A and Topic B" and "Topic C," it can update the multi-data acquired from the second meeting of Topic C by integrating it into the previously stored Topic C meeting minutes (v1.0 (version 1.0)). That is, the electronic device (101) can create and store the Topic C meeting minutes (v1.1 (version 1.1)).

[0234] In summary, when a total of six meetings are held for a total of three topics (topic A, topic B, topic C), the electronic device (101) according to one embodiment can generate three versions of meeting minutes (v1.0, v1.1, and v1.2) for topic A, one meeting minute (v1.0) for topic B, and two meeting minutes (v1.0 and v1.1) for topic C.

[0235] FIG. 12 illustrates an example of an event that can be managed as event data of an electronic device according to one embodiment of the present disclosure.

[0236] FIG. 13 illustrates various examples of event data stored in an electronic device according to one embodiment of the present disclosure.

[0237] Referring to FIGS. 12 and 13, an embodiment for generating event data for events of interest to the user, such as events where the user's continuous query occurs or hobby activities, can be seen.

[0238] Referring to FIGS. 12-A and 12-B, the user can assemble Lego while looking at the instructions (12-C) while wearing a wearable electronic device (e.g., glasses-type electronic device). Additionally, the user can input user queries into the multi-data sensing module (200) during the process of assembling Lego.

[0239] Specifically, by analyzing the multidata acquired from the multidata sensing module (200) of the electronic device (101), "Lego assembly" can be defined as an event based on image data regarding a castle-shaped Lego, Lego assembly instructions, and the user's hand. In addition, "Lego assembly hobby time" can be defined as an event based on the place (e.g., home) and time (e.g., evening after work) where Lego assembly takes place detected by the electronic device (101), Lego box and instructions, and Lego assembly monitoring.

[0240] The electronic device (101) can analyze multi-data acquired from the multi-data sensing module (200) to define "hobby time for assembling Lego" as an event, and store the acquired multi-data as event data for "hobby time for assembling Lego".

[0241] Subsequently, if the user continues to assemble Lego, the electronic device (101) can integrate and store the multi-data obtained from the multi-data sensing module (200) with the previously stored event data.

[0242] For example, the electronic device (101) can generate and provide to the user event data in the form of a brief (summary information) regarding the accumulated stored Lego assembly progress and process. Additionally, when the electronic device (101) receives a query from the user (e.g., Where should I build next?), it can generate and provide a response (e.g., The bottom part of the castle is finished, and it is time to build the tower where Dumbledore's office is located) based on the accumulated generated event data.

[0243] As another example, if a user queries the estimated time to complete a Lego assembly, it is possible to generate and provide a relatively accurate response regarding the remaining amount and estimated time by considering the user's accumulated assembly process and time spent to date.

[0244] That is, the electronic device (101) can show the progress of each type of Lego by date, or create and save the production process story as image data or stop-motion video as event data and provide it to the user (13-A).

[0245] When one type of Lego (e.g., Hogwarts Castle and Garden) is completed, the electronic device (101) can generate and store the Lego assembly progress by date and the production process video (e.g., Hogwarts Castle and Garden production log) as event data and provide it to the user (13-B).

[0246] Additionally, if the electronic device (101) has assembled multiple Lego series during the year, it can display the event data stored for each Lego type together in an associated category so that the user can see them at a glance (13-C). That is, when the user inputs a query regarding "the number of Lego series assembled this year" or "the time taken to assemble a specific Lego series," the electronic device (101) can generate and provide a response based on the event data stored for each type.

[0247] FIG. 14 is a flowchart illustrating a method for generating and providing a response to a user query based on integrated stored event data of an electronic device according to one embodiment of the present disclosure.

[0248] In the following embodiments, each operation of FIG. 14 may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation of FIG. 14 may be changed, and at least two operations may be performed in parallel.

[0249] According to one embodiment, the 4000 to 4200 operations of FIG. 14 can be understood as being performed in a processor (e.g., processor (120) of FIG. 3) of an electronic device (e.g., electronic device (101) of FIG. 3).

[0250] In various embodiments, before performing the 4000 operation, the electronic device (101) may perform an operation to establish communication (e.g., pairing) with an external electronic device (e.g., electronic device (102, 104) of FIG. 1).

[0251] In one embodiment, the processor (120) can receive user query data through an input module (150) (4000). For example, the user can input query data through the input module (150) regarding a "topic to discuss in a meeting" currently being conducted or a "date for a subsequent meeting" based on the minutes of a concluded meeting.

[0252] The processor (120) can define a relevant event from the query data obtained from the input module (150) and search for whether event data related to the query data exists in the electronic device (101) (4100). For example, if a user inputs query data regarding "topics to be discussed at the second patent workshop meeting currently being held," the processor (120) can define "patent workshop" as an event for the user query data and search for whether there is a first patent workshop meeting minutes that is recognized as related to the "patent workshop."

[0253] The processor (120) can generate and provide a response to a user's query based on event data related to the query data (4200). For example, if a user inputs query data regarding "topics to be discussed at the second patent workshop meeting currently being held," the processor (120) recognizes that the meeting currently being held is a subsequent meeting recognized as being related to the "patent workshop," and can generate and provide a response to the user based on the minutes of the first patent workshop meeting existing in the electronic device (101).

[0254] FIGS. 15 and 16 illustrate providing a visual indicator for multidata and event data on a display of an electronic device according to one embodiment of the present disclosure.

[0255] Referring to FIG. 15, according to one embodiment of the present disclosure, a visual indicator is shown on the display of an electronic device (101, e.g., mobile terminal, smartphone) indicating that multi-data acquired from a multi-data sensing module (200) is being stored in a memory (130).

[0256] According to one embodiment, the processor (120) can analyze multi-data acquired from the multi-data sensing module (200) and define a related event. The processor (120) can store the acquired multi-data as event data for the defined event. At this time, the electronic device (101) can provide an indicator (e.g., Recording in progress) to notify the user that the multi-data is being stored as event data.

[0257] According to one embodiment, the indicator may include not only a visual indicator but also a voice indicator. The visual indicator may be displayed on the display module (160) of the electronic device (101) (15-A). The voice indicator may be output as voice data through the sound output module (155). However, it is not limited thereto.

[0258] Referring to FIG. 16, according to one embodiment of the present disclosure, a visual indicator (16-A) is shown on an electronic device (101, e.g., glasses-type wearable electronic device) indicating that multi-data acquired from a multi-data sensing module (200) is being stored in a memory (130).

[0259] According to one embodiment, the processor (120) can analyze multi-data acquired from the multi-data sensing module (200) to define a related event (e.g., patent workshop). The processor (120) can store the acquired multi-data as event data for the defined event (e.g., patent workshop meeting minutes). At this time, an indicator (e.g., Recording in progress) can be provided to notify the user that the multi-data is being stored as event data. Additionally, the indicator may include not only information (e.g., Recording in progress) indicating that the electronic device (101) is currently storing the multi-data as event data, but also event information defined from the multi-data (e.g., Patent meeting recording in progress). However, the information that the indicator may include is not limited thereto.

[0260] FIG. 17 illustrates an example of a plurality of electronic devices capable of acquiring multiple data for a single event data according to one embodiment of the present disclosure.

[0261] Referring to FIG. 17, multiple electronic devices that acquire multi-data recognized as relevant to a specific event can be seen. Additionally, the multiple electronic devices may be electronic devices to which a separate authorization is granted to allow a user to acquire multi-data regarding a specific event. The multiple electronic devices (101) may include smartphones, wearable devices (e.g., head-mounted devices or glasses-type devices), tablet PCs, smart TVs, robots, and drones.

[0262] For example, if a meeting regarding a "patent workshop" is currently in progress, multi-data regarding the patent workshop meeting may be obtained through multiple electronic devices (17-A to 17-F) present in the meeting room. However, according to one embodiment, the multiple electronic devices through which multi-data regarding the patent workshop meeting is obtained may be some electronic devices authorized by a user (e.g., 17-A, 17-E, 17-F).

[0263] An electronic device according to one embodiment of the present disclosure comprises a multi-data sensing module, an event data output module, a memory for storing instructions, and at least one processor. When the instructions are executed by the at least one processor, the electronic device can acquire multi-data from the multi-data sensing module. When the instructions are executed by the at least one processor, the electronic device can analyze the multi-data to define an event. When the instructions are executed by the at least one processor, the electronic device can determine whether event data related to the event exists. When the instructions are executed by the at least one processor, if the event data exists, the electronic device can integrate and store the multi-data into the event data. When the instructions are executed by the at least one processor, the electronic device can provide the integrated and stored event data through the event data output module.

[0264] According to one embodiment, the multidata is a first multidata, the event is a first event, and the event data associated with the first event is the first event data, and when the instructions are executed by the at least one processor, the electronic device may store the first multidata as the first event data if the first event data does not exist. When the instructions are executed by the at least one processor, the electronic device may provide the first event data through the event data output module.

[0265] According to one embodiment, the event is a first event, the event data associated with the first event is the first event data, the integrated stored event data is the first-1 event data, and the instructions, when executed by the at least one processor, enable the electronic device to reacquire multi-data from the multi-data sensing module. When the instructions are executed by the at least one processor, if the reacquired multi-data is associated with the first event, the electronic device can integrate and store the reacquired multi-data in the first-1 event data.

[0266] According to one embodiment, the event is a first event, the event data related to the first event is the first event data, the integrated stored event data is the first-1 event data, and the instructions, when executed by the at least one processor, enable the electronic device to re-acquire multi-data from the multi-data sensing module. When the instructions, when executed by the at least one processor, enable the electronic device to define a second event by analyzing the re-acquired multi-data if the re-acquired multi-data is not related to the first event. When the instructions, when executed by the at least one processor, enable the electronic device to determine whether there is second event data related to the second event. When the instructions, when executed by the at least one processor, enable the electronic device to integrate and store the re-acquired multi-data into the second event data if the second event data exists.

[0267] According to one embodiment, when the instructions are executed by the at least one processor, the electronic device can analyze common data information between the multi-data and the event data. When the instructions are executed by the at least one processor, the electronic device can summarize the integrated stored event data based on the analyzed data information. When the instructions are executed by the at least one processor, the electronic device can provide the summarized event data.

[0268] According to one embodiment, when the instructions are executed by the at least one processor, the electronic device can receive query data from a user. When the instructions are executed by the at least one processor, the electronic device can determine whether event data related to the query data exists. When the instructions are executed by the at least one processor, the electronic device can provide a response to the query data to the user based on the event data if the event data exists.

[0269] According to one embodiment, the event data output module includes a display module, and when the instructions are executed by the at least one processor, the electronic device can provide a visual indicator to a user through the display that the multi-data is integrated and stored in the event data.

[0270] According to one embodiment, the multi-data sensing module includes a camera module and an audio module, and the multi-data includes at least one of an image, video, or text obtained from the camera module, and may include audio data obtained from the audio module.

[0271] According to one embodiment, the event may include an event related to multi-data that the user of the electronic device intends to track and manage among the multi-data acquired from the multi-data sensing module.

[0272] According to one embodiment, the electronic device further includes a communication module, and when the instructions are executed by the at least one processor, the electronic device can obtain the multidata from at least one external electronic device communicating through the communication module.

[0273] According to one embodiment, the at least one external electronic device may include at least one external electronic device authorized by a user in response to the event.

[0274] A method for providing event data of an electronic device comprising a multi-data sensing module, an event data output module, and a memory according to one embodiment of the present disclosure may include an operation of acquiring multi-data from the multi-data sensing module. The providing method may include an operation of defining an event by analyzing the multi-data. The providing method may include an operation of determining whether there is event data related to the event. If the event data exists, the providing method may include an operation of integrating the multi-data into the event data and storing it. The providing method may include an operation of providing the integrated and stored event data through the event data output module.

[0275] According to one embodiment, the multidata is a first multidata, the event is a first event, and the event data associated with the first event is the first event data, and the method may include an operation of storing the first multidata as the first event data when the first event data does not exist. The method may further include an operation of providing the first event data through the event data output module.

[0276] According to one embodiment, the method comprises: the event is a first event; event data related to the first event is the first event data; and the integrated stored event data is the first-1 event data. The method may include an operation of reacquiring multi-data from the multi-data sensing module. The method may further include an operation of integrating and storing the reacquired multi-data into the first-1 event data when the reacquired multi-data is related to the first event.

[0277] According to one embodiment, the event is a first event, the event data related to the first event is the first event data, the integrated stored event data is the first-1 event data, and the method may include an operation of reacquiring multi-data from the multi-data sensing module. If the reacquired multi-data is not related to the first event, the method may include an operation of analyzing the reacquired multi-data to define a second event. The method may include an operation of determining whether there is a second event data related to the second event. If the second event data exists, the method may further include an operation of integrating the reacquired multi-data into the second event data and storing it.

[0278] According to one embodiment, the method may include an operation of analyzing common data information between the multi-data and the event data. The method may further include an operation of summarizing the integrated stored event data based on the analyzed data information and an operation of providing the summarized event data.

[0279] According to one embodiment, the method may include an operation of receiving query data from a user. The method may include an operation of determining whether event data related to the query data exists. If the event data exists, the method may further include an operation of providing a response to the query data to the user based on the event data.

[0280] According to one embodiment, the event data output module includes a display module, and the method may further include an operation of providing a user with a visual indicator through the display that the multi-data is integrated and stored in the event data.

[0281] According to one embodiment, the multi-data sensing module includes a camera module and an audio module, and the multi-data includes at least one of an image, video, or text obtained from the camera module, and may include audio data obtained from the audio module.

[0282] According to one embodiment, the event may include an event related to multi-data that the user of the electronic device intends to track and manage among the multi-data acquired from the multi-data sensing module.

[0283] According to one embodiment, the method may further include a communication module, and the method may further include the operation of acquiring the multidata from at least one external electronic device communicating through the communication module.

[0284] According to one embodiment, the at least one external electronic device may include at least one external electronic device authorized by a user in response to the event.

[0285] An electronic device according to one embodiment of the present disclosure can provide a method and device capable of efficiently recording and classifying a vast amount of data by continuously managing data recognized as relevant by integrating and storing multi-data when event data related to multi-data acquired from a data sensing module exists.

[0286] An electronic device according to one embodiment of the present disclosure can provide a method and device that can enhance user convenience by summarizing continuously managed data to provide information to a user in the form of a simple brief, and by generating and providing an optimal response that considers the overall context when a user query is input that is recognized as relevant to the continuously managed data.

[0287] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0288] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as "coupled" or "connected" to another (e.g., 2nd) component, with or without the terms "functionally" or "communicationly," it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0289] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0290] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (36) or external memory (38)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0291] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0292] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0293] The embodiments disclosed in this specification and drawings are merely specific examples provided to facilitate the explanation of the technical content according to the embodiments of this disclosure and to aid in understanding the embodiments of this disclosure, and are not intended to limit the scope of the embodiments of this disclosure. Accordingly, the scope of an embodiment of this disclosure should be interpreted as including all modifications or variations derived based on the technical features of an embodiment of this disclosure, in addition to the embodiments disclosed herein.

Claims

1. In an electronic device (101), Multi-data sensing module (200); Event data output module (210); Memory (130) for storing instructions; and It includes at least one processor (120); and When the above instructions are executed by the at least one processor, the electronic device, Acquire multidata from the above multidata sensing module, and Define events by analyzing the above multi-data, and Determine whether there is event data related to the above event, and If the above event data exists, the above multi-data is integrated into the above event data and stored, and An electronic device that provides the integrated stored event data through the above event data output module.

2. In Paragraph 1, The above multidata is the first multidata, and The above event is the first event, and The event data related to the above first event is the first event data, and When the above instructions are executed by the at least one processor, the electronic device, If the above first event data does not exist, the above first multi-data is stored as the first event data, and An electronic device that provides the first event data through the above event data output module.

3. In Paragraph 1, The above event is the first event, and The event data related to the above first event is the first event data. The above integrated stored event data is the 1-1 event data, and When the above instructions are executed by the at least one processor, the electronic device, Reacquiring multidata from the above multidata sensing module, and An electronic device that integrates and stores the reacquired multi-data into the first-1 event data when the reacquired multi-data is related to the first event.

4. In Paragraph 1, The above event is the first event, and The event data related to the above first event is the first event data. The above integrated stored event data is the 1-1 event data, and When the above instructions are executed by the at least one processor, the electronic device, Reacquiring multidata from the above multidata sensing module, and If the reacquired multi-data above is not related to the first event, the reacquired multi-data above is analyzed to define a second event, and Determining whether there is second event data related to the above second event, and An electronic device that integrates and stores the reacquired multi-data into the second event data when the second event data exists.

5. In Paragraph 1, When the above instructions are executed by the at least one processor, the electronic device, Analyze common data information between the above multi-data and the above event data, and Based on the above-mentioned analyzed data information, the above-mentioned integrated stored event data is summarized, and An electronic device that provides the above-summarized event data.

6. In Paragraph 1, When the above instructions are executed by the at least one processor, the electronic device, Receive query data from the user, Determine whether event data related to the above query data exists, and An electronic device that, when the above event data exists, provides a response to the above query data to the user based on the above event data.

7. In Paragraph 1, The above event data output module (210) includes a display module (160), and When the above instructions are executed by the at least one processor, the electronic device, An electronic device that provides a visual indicator to the user, via the display, that the multi-data is integrated and stored in the event data.

8. In Paragraph 1, The above multi-data sensing module (200) includes a camera module (180) and an audio module (170), and The above multidata includes at least one of an image, video, or text obtained from the camera module and audio data obtained from the audio module, and The above event is an electronic device comprising an event related to multidata that the user of the electronic device intends to track and manage among the multidata acquired from the multidata sensing module.

9. In Paragraph 1, Including a communication module; further including; When the above instructions are executed by the at least one processor, the electronic device, An electronic device that communicates through the communication module and acquires the multidata from at least one external electronic device authorized by the user in response to the event.

10. A method for providing event data of an electronic device comprising a multi-data sensing module (200), an event data output module (210), and a memory (130), Operation of acquiring multidata from the above multidata sensing module; An operation to define an event by analyzing the above multi-data; An operation to determine whether event data related to the above event exists; If the above event data exists, the operation of integrating the multi-data into the event data and storing it; and A method including the operation of providing the integrated stored event data through the above event data output module.

11. In Paragraph 10, The above multidata is the first multidata, and The above event is the first event, and The event data related to the above first event is the first event data, and The above method is, If the above first event data does not exist, the operation of storing the above first multi-data as the first event data; and A method further comprising the operation of providing the first event data through the above event data output module.

12. In Paragraph 10, The above event is the first event, and The event data related to the above first event is the first event data, and The above integrated stored event data is the 1-1 event data, and The above method is, The operation of reacquiring multidata from the above multidata sensing module; and A method further comprising the operation of integrating and storing the reacquired multi-data into the 1-1 event data when the reacquired multi-data is related to the 1-1 event.

13. In Paragraph 10, The above event is the first event, and The event data related to the above first event is the first event data, and The above integrated stored event data is the 1-1 event data, and The above method is, Operation of reacquiring multi-data from the above multi-data sensing module; If the reacquired multi-data is not related to the first event, the operation of analyzing the reacquired multi-data to define a second event; An operation to determine whether there is second event data related to the second event; and A method further comprising, when the above second event data exists, integrating the reacquired multi-data into the above second event data and storing it.

14. In Paragraph 10, The above method is, An operation to analyze common data information between the above multi-data and the above event data; An operation to summarize the integrated stored event data based on the analyzed data information above; and A method further including the operation of providing the above-summarized event data.

15. In Paragraph 10, The above method is, Operation of receiving query data from the user; An operation to determine whether event data related to the above query data exists; and A method further comprising, when the above event data exists, providing a response to the above query data to the user based on the above event data.

Images