Artificial intelligence device and operating method thereof

Abstract

An artificial intelligence device according to an embodiment of the present disclosure may comprise: a display; a communication interface that communicates with a server; a memory storing an artificial intelligence model that infers a content recognition result on the basis of at least one of video, audio, or text output from the display; and a processor that acquires viewing history information including the content recognition result, generates obfuscation data from the viewing history information by using an obfuscation technique, and transmits the generated obfuscation data to the server via the communication interface.

Description

Artificial intelligence device and its method of operation

[0001] The present invention relates to an artificial intelligence device, and more specifically, to an artificial intelligence device capable of providing personalized services.

[0002] Automatic Content Recognition (ACR) technology is a technology that automatically recognizes and analyzes content such as broadcasts, videos, and music. Through ACR technology, TV contributes to enhancing the user experience by providing real-time viewing information, targeted advertisements, and additional information.

[0003] However, the recognition targets of conventional ACR technology are limited to broadcast or terrestrial content, and follow a linear transmission method in which the transmission target content, transmission time, and transmission area are standardized.

[0004] In addition, conventional ACR technology-based services had problems such as excessive transmission of content data, limited interactive services, and increased operating costs due to the presence of servers for fingerprints and watermarks.

[0005] Furthermore, in the case of conventional ACR technology-based services, viewing information is transmitted to the server based on asymmetric encryption, so there is a risk that personal information, such as viewing data, may be exposed.

[0006] The purpose of the present disclosure may be to expand the coverage of existing ACR technology to provide more specialized and customized services.

[0007] The purpose of the present disclosure may be to reduce the data loading burden on the server by transmitting only the minimum amount of data for user viewing analysis to the server.

[0008] The purpose of the present disclosure may be to prevent the risk of exposure of personal information by obfuscating viewing information.

[0009] The purpose of the present disclosure may be to analyze a user's content consumption patterns and provide targeted advertisements and recommended content specialized for the user.

[0010] An artificial intelligence device according to one embodiment of the present disclosure may include: a display; a communication interface for communicating with a server; a memory for storing an artificial intelligence model that infers a content recognition result based on one or more of an image, audio, or text output from the display; and a processor that acquires viewing history information including the content recognition result, generates obfuscated data from the viewing history information using an obfuscation technique, and transmits the generated obfuscated data to the server through the communication interface.

[0011] A method of operation of an artificial intelligence device according to one embodiment of the present disclosure may include: outputting one or more of video, audio, or text; inferring a content recognition result based on one or more of the video, audio, or text output through an artificial intelligence model; obtaining viewing history information including the content recognition result; generating obfuscated data from the viewing history information using an obfuscation technique; and transmitting the generated obfuscated data to a server.

[0012] According to an embodiment of the present disclosure, a personalized service may be provided to each of a plurality of users.

[0013] In addition, according to an embodiment of the present disclosure, the privacy of user data transmitted externally can be protected, thereby reducing damage from data leakage.

[0014] In addition, according to an embodiment of the present disclosure, a data server can perform various statistical analyses based on parametric analysis even when receiving modified numerical data (obfuscated data).

[0015] In addition, according to an embodiment of the present disclosure, content can be recognized on a non-linear transmission platform, user interests can be derived to record viewing history, and proactive customer information can be built.

[0016] FIG. 1 is a block diagram for explaining the components of an artificial intelligence device according to one embodiment of the present disclosure.

[0017] FIG. 2 is a drawing for explaining the configuration of an artificial intelligence server according to one embodiment of the present disclosure.

[0018] FIG. 3 is a drawing for explaining the configuration of a system according to one embodiment of the present disclosure.

[0019] FIG. 4 is a sequence diagram for explaining the operation method of a system according to one embodiment of the present disclosure.

[0020] FIG. 5 is a diagram illustrating the process of obtaining content recognition results through artificial intelligence models according to one embodiment of the present disclosure.

[0021] FIGS. 6a to 6c are drawings illustrating examples of artificial intelligence models according to embodiments of the present disclosure.

[0022] FIG. 7a is a flowchart illustrating the process of generating obfuscated data from viewing history information through an obfuscation technique according to one embodiment of the present disclosure.

[0023] FIG. 7b is a flowchart illustrating the process of updating an obfuscated argument set according to one embodiment of the present disclosure.

[0024] Figures 8a and 8b are figures comparing the obfuscation results of original data according to the degree of obfuscation.

[0025] FIG. 9 is a diagram illustrating an example of providing a personalized service for each user according to an embodiment of the present disclosure.

[0026] FIG. 10 is a drawing for explaining the configuration of a data server according to one embodiment of the present disclosure.

[0027] FIG. 11 is a drawing for explaining the configuration of a data server according to one embodiment of the present disclosure.

[0028] Artificial intelligence refers to the field of researching artificial intelligence or the methodologies to create it, while machine learning refers to the field of researching methodologies to define the various problems addressed in the field of artificial intelligence and to solve them.

[0029] Machine learning is also defined as an algorithm that improves the performance of a task through consistent experience.

[0030] An Artificial Neural Network (ANN) is a model used in machine learning that can refer to a model capable of problem-solving, composed of artificial neurons (nodes) that form a network through the connection of synapses.

[0031] An artificial neural network can be defined by connection patterns between neurons in different layers, a learning process that updates model parameters, and an activation function that generates output values.

[0032] An artificial neural network may include an input layer, an output layer, and optionally one or more hidden layers. Each layer may include one or more neurons, and the artificial neural network may include synapses connecting the neurons. In an artificial neural network, each neuron may output a function value of an activation function for input signals, weights, and biases input through the synapses.

[0033] Model parameters refer to parameters determined through learning, including synaptic connection weights and neuron biases. Hyperparameters refer to parameters that must be set before training in a machine learning algorithm, including the learning rate, number of iterations, mini-batch size, and initialization function.

[0034] The objective of training an artificial neural network can be viewed as determining model parameters that minimize the loss function. The loss function can be used as an indicator to determine optimal model parameters during the training process of an artificial neural network.

[0035] Machine learning can be classified into supervised learning, unsupervised learning, and reinforcement learning depending on the learning method.

[0036] Supervised learning refers to a method of training an artificial neural network with labels provided for the training data; a label can refer to the correct answer (or result value) that the artificial neural network must infer when training data is input into it.

[0037] Unsupervised learning can refer to a method of training an artificial neural network without being given labels for the training data.

[0038] Reinforcement learning can refer to a learning method that trains an agent defined within an environment to select an action or sequence of actions that maximizes the cumulative reward in each state.

[0039] Machine learning implemented as a deep neural network (DNN) containing multiple hidden layers among artificial neural networks is also called deep learning, and deep learning is a part of machine learning.

[0040] In the following, machine learning is used to include deep learning.

[0041] FIG. 1 is a block diagram for explaining the components of an artificial intelligence device according to one embodiment of the present disclosure.

[0042] The artificial intelligence device (100) can be implemented as a stationary device or a mobile device, such as a TV, projector, mobile phone, smartphone, desktop computer, laptop, digital broadcasting terminal, PDA (personal digital assistants), PMP (portable multimedia player), navigation, tablet PC, wearable device, set-top box (STB), DMB receiver, radio, washing machine, refrigerator, desktop computer, digital signage, robot, vehicle, etc.

[0043] Referring to FIG. 1, the artificial intelligence device (100) may include a communication interface (110), an input interface (120), a learning processor (130), a sensor (140), an output interface (150), a memory (170), and a processor (180).

[0044] The communication interface (110) can transmit and receive data with external devices, such as other artificial intelligence devices or AI servers (200), using wired or wireless communication technology. For example, the communication interface (110) can transmit and receive sensor information, user input, learning models, control signals, etc., with external devices.

[0045] The communication technologies used by the communication interface (110) include GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), LTE (Long Term Evolution), 5G, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Bluetooth (Bluetooth), RFID (Radio Frequency Identification), Infrared Data Association (IrDA), ZigBee, NFC (Near Field Communication), etc.

[0046] The input interface (120) can acquire various types of data.

[0047] The input interface (120) may include a camera (121) for capturing images, a microphone (122) for receiving audio signals, and a user input interface (123) for receiving information from a user.

[0048] A camera (121) or a microphone (122) can be treated as a sensor, and a signal obtained from the camera (121) or the microphone (122) can be named as sensing data or sensor information.

[0049] The input interface (120) can obtain input data to be used when obtaining an output using training data and a training model for model training. The input interface (120) may also obtain unprocessed input data, in which case the processor (180) or the learning processor (130) can extract input features as a preprocessing step for the input data.

[0050] The camera (121) processes image frames, such as still images or video, obtained by an image sensor in video call mode or shooting mode. The processed image frames may be displayed on a display (151) or stored in memory (170).

[0051] The microphone (122) processes external acoustic signals into electrical voice data. The processed voice data can be utilized in various ways depending on the function (or application running) being performed on the artificial intelligence device (100). Meanwhile, various noise removal algorithms can be applied to the microphone (122) to remove noise generated during the process of receiving external acoustic signals.

[0052] The user input interface (123) is for receiving information from a user, and when information is input through the user input interface (123), the processor (180) can control the operation of the artificial intelligence device (100) to correspond to the input information.

[0053] The user input interface (123) may include mechanical input means (or mechanical keys, such as buttons, dome switches, jog wheels, jog switches, etc. located on the front / rear or side of the artificial intelligence device (100)) and touch input means.

[0054] As an example, a touch input means may consist of a virtual key, soft key, or visual key displayed on a touchscreen through software processing, or a touch key placed on a part other than the touchscreen.

[0055] The learning processor (130) can train a model composed of an artificial neural network using training data. The trained artificial neural network can be called a learning model. The learning model can be used to infer a result value for new input data other than the training data, and the inferred value can be used as a basis for judgment to perform some action.

[0056] The learning processor (130) can perform AI processing together with the learning processor (240) of the AI ​​server (200).

[0057] The learning processor (130) may include memory integrated into or implemented in the artificial intelligence device (100). The learning processor (130) may also be implemented using memory (170), external memory directly coupled to the artificial intelligence device (100), or memory maintained in an external device.

[0058] The sensor (140) can acquire at least one of internal information of the artificial intelligence device (100), surrounding environment information of the artificial intelligence device (100), and user information using various sensors.

[0059] The sensor (140) may include one or more of a proximity sensor, an illuminance sensor, an accelerometer, a magnetic sensor, a gyroscope, an inertial sensor, an RGB sensor, an IR sensor, a fingerprint recognition sensor, an ultrasonic sensor, a light sensor, a microphone, a lidar sensor, and a radar sensor.

[0060] The output interface (150) can generate output related to visual, auditory, or tactile senses.

[0061] The output interface (150) may include a display (151) for outputting an image, an audio output interface (152) for outputting audio, a haptic device (153) for outputting tactile information, and a light output interface (154) for outputting light.

[0062] The display (151) displays (outputs) information processed by the artificial intelligence device (100). For example, the display (151) can display information on the execution screen of an application running on the artificial intelligence device (100), or UI (User Interface) and GUI (Graphic User Interface) information based on such execution screen information.

[0063] The display (151) can be implemented as a touch screen by forming a layered structure with the touch sensor or by being formed as an integral unit. The touch screen functions as a user input interface (123) that provides an input interface between the artificial intelligence device (100) and the user, and at the same time, can provide an output interface between the artificial intelligence device (100) and the user.

[0064] The audio output interface (152) can output audio data received from the communication interface (110) or stored in the memory (170) in call signal reception, call mode or recording mode, voice recognition mode, broadcast reception mode, etc.

[0065] The audio output interface (152) may include at least one of a receiver, a speaker, and a buzzer.

[0066] The haptic device (153) generates various tactile effects that the user can feel. A typical example of the tactile effect generated by the haptic device (153) can be vibration.

[0067] The light output interface (154) outputs a signal to indicate the occurrence of an event using the light of the light source of the artificial intelligence device (100). Examples of events occurring in the artificial intelligence device (100) may include receiving a message, receiving a call signal, a missed call, an alarm, a schedule notification, receiving an email, receiving information through an application, etc.

[0068] The memory (170) can store data that supports various functions of the artificial intelligence device (100). For example, the memory (170) can store input data, learning data, learning models, learning history, etc. obtained from the input interface (120).

[0069] The processor (180) can determine at least one executable action of the artificial intelligence device (100) based on information determined or generated using a data analysis algorithm or a machine learning algorithm.

[0070] The processor (180) can control the components of the artificial intelligence device (100) to perform a determined operation.

[0071] To this end, the processor (180) can request, search, receive, or utilize data from the learning processor (130) or memory (170), and can control the components of the artificial intelligence device (100) to execute a predicted operation or a preferred operation among the at least one executable operation.

[0072] If the processor (180) requires the connection of an external device to perform a determined operation, it can generate a control signal to control the external device and transmit the generated control signal to the external device.

[0073] The processor (180) can obtain intent information regarding user input and determine the user's requirements based on the obtained intent information.

[0074] The processor (180) can obtain intent information corresponding to user input by using at least one of a Speech To Text (STT) engine for converting voice input into a string or a Natural Language Processing (NLP) engine for obtaining intent information of natural language.

[0075] At least one of the STT engine or NLP engine may be composed of an artificial neural network, at least a portion of which is trained according to a machine learning algorithm. Additionally, at least one of the STT engine or NLP engine may be trained by a learning processor (130), trained by a learning processor (240) of an AI server (200), or trained by distributed processing thereof.

[0076] The processor (180) can collect history information, including the operation details of the artificial intelligence device (100) or user feedback regarding the operation, and store it in memory (170) or a learning processor (130), or transmit it to an external device such as an AI server (200). The collected history information can be used to update a learning model.

[0077] The processor (180) can control at least some of the components of the artificial intelligence device (100) to run an application stored in memory (170).

[0078] The processor (180) can operate two or more of the components included in the artificial intelligence device (100) in combination with each other for the operation of the above application.

[0079] FIG. 2 is a drawing for explaining the configuration of an artificial intelligence server according to one embodiment of the present disclosure.

[0080] Referring to FIG. 2, the AI ​​server (200) may refer to a device that trains an artificial neural network using a machine learning algorithm or uses a trained artificial neural network.

[0081] The AI ​​server (200) may be composed of multiple servers to perform distributed processing and may be defined as a 5G network. The AI ​​server (200) may be included as part of the configuration of the artificial intelligence device (100) to perform at least some of the AI ​​processing together.

[0082] The AI ​​server (200) may include a communication interface (210), memory (230), a learning processor (240), and a processor (260).

[0083] The communication interface (210) can transmit and receive data with an external device such as an artificial intelligence device (100).

[0084] The memory (230) may include a model memory (231). The model memory (231) may store a model (or artificial neural network, 231a) that is being learned or has been learned through the learning processor (240).

[0085] The learning processor (240) can train the artificial neural network (231a) using training data. The training model may be used while mounted on the AI ​​server (200) of the artificial neural network, or it may be used while mounted on an external device such as an artificial intelligence device (100).

[0086] The learning model may be implemented in hardware, software, or a combination of hardware and software. If part or all of the learning model is implemented in software, one or more instructions constituting the learning model may be stored in memory (230).

[0087] The processor (260) can use a learning model to infer a result value for new input data and generate a response or control command based on the inferred result value.

[0088] FIG. 3 is a drawing for explaining the configuration of a system according to one embodiment of the present disclosure.

[0089] Referring to FIG. 3, the system (30) may include an artificial intelligence device (100), a data server (200-1), and a business server (300).

[0090] The data server (200-1) may be an example of the AI ​​server (200) of FIG. 2. The data server (200-1) may include all components of the AI ​​server (200).

[0091] The artificial intelligence device (100) can collect content information about the content video.

[0092] The artificial intelligence device (100) can obtain viewing history information including content recognition results based on collected content information.

[0093] The artificial intelligence device (100) can generate obfuscated data from viewing history information through an obfuscation technique.

[0094] The artificial intelligence device (100) can transmit obfuscated data to the data server (200-1).

[0095] The data server (200-1) can obtain a viewing analysis result based on the received obfuscated data.

[0096] The data server (200-1) can transmit the results of the viewing analysis to the operator server (300).

[0097] The operator server (300) can obtain customized content information based on the received viewing analysis results and can transmit the obtained customized content information to the data server (200-1).

[0098] The data server (200-1) can transmit customized content information received from the operator server (300) to the artificial intelligence device (100).

[0099] The artificial intelligence device (100) can output customized content information.

[0100] In the following, viewing history information and obfuscated data based on viewing history information may be private data requiring user privacy.

[0101] Obfuscated argument sets, custom content information, and new labeling data may be public data where user privacy is not required.

[0102] FIG. 4 is a sequence diagram for explaining the operation method of a system according to one embodiment of the present disclosure.

[0103] Referring to FIG. 4, the processor (180) of the artificial intelligence device (100) can collect content information about a content video being played on a display (151) (S401).

[0104] Content information may include one or more of images extracted from the content video, audio, or subtitles of the content video. The images extracted from the content video may be captured images that constitute a part of the content video.

[0105] The processor (180) can periodically collect content information about the content video that the user is watching.

[0106] The processor (180) stores video or audio output from the display (151) in a buffer as soon as power is applied to the artificial intelligence device (100), and can consider the sequence of the stored buffer as a media stream.

[0107] The processor (180) of the artificial intelligence device (100) can obtain viewing history information including content recognition results based on collected content information (S403).

[0108] The processor (180) can obtain content recognition results from content information through an artificial intelligence model stored in memory (170).

[0109] The content recognition result may include at least one of the name of an object included in the scene of the content video, the situation of the scene, the content of the scene, the genre of the content video, or the name of the content video.

[0110] For example, an artificial intelligence model may be a model that outputs a content recognition result from one or more of an image, audio, or subtitle extracted from a content video.

[0111] As another example, an artificial intelligence model may be a model that infers content recognition results based on input and reference information.

[0112] Multiple artificial intelligence models can be used to obtain content recognition results.

[0113] The artificial intelligence model may include at least one model among a natural language processing model or a behavioral recognition model learned through a deep learning algorithm or a machine learning algorithm.

[0114] FIG. 5 is a diagram illustrating the process of obtaining content recognition results through artificial intelligence models according to one embodiment of the present disclosure.

[0115] Referring to FIG. 5, a natural language processing model (510) and a behavior recognition model (520) stored in memory (170) are shown.

[0116] The processor (180) of the artificial intelligence device (100) can capture a content video to obtain a captured image and can extract text from the captured image using an Optical Character Recognition (OCR) technique.

[0117] The natural language processing model (510) can output a first inference result including a first valid viewing interval and a first content from text extracted using a natural language processing engine. The first content may include the theme of the content video, keywords, and the emotions of the person.

[0118] The processor (180) of the artificial intelligence device (100) can generate a captured video based on a plurality of captured images captured from a content video. The processor (180) can generate a captured video having a length of 10 seconds using the captured images after capturing 5 times per second.

[0119] The behavior recognition model (520) can infer a second valid viewing interval and second content from the captured video. The behavior recognition model (520) can infer a second valid viewing interval and second content of the content video by recognizing the behavior of the object.

[0120] The behavior recognition model (520) can output a second inference result from a captured image using an attention technique and a transformer technique.

[0121] The processor (180) can generate a content recognition result based on the first inference result and the second inference result. The processor (180) can determine the portion where the first valid viewing section and the second valid viewing section overlap as the final valid viewing section.

[0122] The processor (180) can determine the overlapping content between the first content and the second content as the final content. The processor (180) can obtain at least one of the final valid viewing interval or the final content as a content recognition result.

[0123] FIGS. 6a to 6c are drawings illustrating examples of artificial intelligence models according to embodiments of the present disclosure.

[0124] Each artificial intelligence model can be stored in memory (170).

[0125] Artificial intelligence models can analyze input information and return output information based on reference information.

[0126] An artificial intelligence model can perform a process of analyzing input information, sorting reference information in batches based on reference information, and returning a portion of it.

[0127] An artificial intelligence model can perform a process of analyzing input information and recording and returning the correlation of all reference information based on reference information.

[0128] The artificial intelligence model may be a model based on an Attention / Transformer layer and a multi-layered artificial neural network structure.

[0129] The input information may include at least one of a video stream, an audio stream, or a text stream extracted from the content image. The input information may further include inference results derived at a past point in time.

[0130] Reference information may be labeling data (or ground truth data). Reference information may represent any identifier, such as keywords or classes. Reference information may be hierarchical according to the semantic construction of the identifiers.

[0131] The output information may be information inferred by the artificial intelligence model.

[0132] The first artificial intelligence model (610) may be a model that recognizes objects from a video stream extracted from a content video (601). The video stream may be the captured video. The labeling data of the first artificial intelligence model (610) may be the name of the object. The name of the object may be any one of dog, lion, bird, others.

[0133] The first artificial intelligence model (610) can output the probability of being classified by the name of each object. The processor (180) can identify the name of the object corresponding to the largest probability among the output probabilities.

[0134] The second artificial intelligence model (620) may be a model that infers the content of the content video (602) from video streams and text streams extracted from the content video (602). The labeling data of the second artificial intelligence model (620) may indicate the type of exercise. The type of exercise may be any one of soccer, baseball, exercise, and others.

[0135] The second artificial intelligence model (620) can output a probability of being classified into each type of motion. The processor (180) can identify the type of motion corresponding to the largest probability among the output probabilities.

[0136] The third artificial intelligence model (630) may be a model that infers the content of a content video from video streams and text streams extracted from the content video. The labeling data of the third artificial intelligence model (630) may include more “home yoga” compared to the labeling data of the second artificial intelligence model (620).

[0137] The third artificial intelligence model (630) may be a model trained with additional labeling data compared to the second artificial intelligence model (620). The new labeling data may be received from the data server (200-1).

[0138] Again, Figure 4 is explained.

[0139] Viewing history information may include content recognition results.

[0140] In addition to the content recognition results, the viewing history information may further include at least one of the following: the genre of the content video watched by the user by time of day, the number of times a specific genre is watched by time of day, the viewing time or valid viewing interval of each content video, a summary of the content video, object objects, person objects, business logos, trademarks, or phone numbers.

[0141] Viewing history information may include private data representing the user's personal information. The private data may include data in the form that the user claims as their own data, and data transmitted externally from the artificial intelligence device (100) when identifying the user.

[0142] Private data may include interests reflecting the user's viewing intention or behavioral history, lists of keywords, lists indicating preferred genres, viewing times of genres by date / time, or information that can be combined with or derived from information identifying the user along with the examples described above.

[0143] Viewing history information may consist of some or all of private data.

[0144] The processor (180) of the artificial intelligence device (100) can generate obfuscated data from viewing history information through an obfuscation technique (S405).

[0145] In one embodiment, the obfuscation technique may be a technique that makes the original content difficult to understand in order to protect data. The obfuscation technique may be utilized to maintain the usefulness of data while protecting sensitive information.

[0146] In one embodiment, the obfuscation technique may be a Local Differential Privacy technique. A Local Differential Privacy technique may be a technique in which the user protects privacy by obfuscating the viewing history information themselves before transmitting the viewing history information to the data server (200-1).

[0147] FIG. 7a is a flowchart illustrating the process of generating obfuscated data from viewing history information through an obfuscation technique according to one embodiment of the present disclosure, and FIG. 7b is a flowchart illustrating the process of updating an obfuscated argument set according to one embodiment of the present disclosure.

[0148] FIG. 7a may be a drawing specifically illustrating step S405 of FIG. 4.

[0149] In the following, the original data may be viewing history information.

[0150] The processor (180) of the artificial intelligence device (100) can obtain an obfuscated argument set (S701).

[0151] The set of obfuscation factors may include a first obfuscation factor (ε) representing the degree of obfuscation, a second obfuscation factor (δ) representing the statistical significance range, and a third obfuscation factor (K) representing the minimum number of labels for verifying the obfuscation result.

[0152] The artificial intelligence device (100) can receive a set of obfuscation arguments from the data server (200-1). The set of obfuscation arguments may vary depending on the type of original data. The original data may be referred to as viewing history information. The data server (200-1) can transmit a first set of obfuscation arguments to the artificial intelligence device (100) for the original data of the first type, and a second set of obfuscation arguments to the artificial intelligence device (100) for the original data of the second type.

[0153] The obfuscated argument set can be a type of public data.

[0154] Multiple obfuscation factors included in the set of obfuscation factors may include a common obfuscation factor applicable to all types of original data and an individual obfuscation factor applicable to a specific type of original data. At least one of the first obfuscation factor (ε), the second obfuscation factor (δ), and the third obfuscation factor (K) may be a common obfuscation factor, and at least one may be an individual obfuscation factor.

[0155] For example, the first obfuscation factor (ε) may be a common obfuscation factor, and the second obfuscation factor (δ) and the third obfuscation factor (K) may be individual obfuscation factors.

[0156] The processor (180) may request a set of obfuscation arguments that matches the type of original data from the data server (200-1). For example, if the processor (180) collects original data of the first type and receives a second set of obfuscation arguments from the data server (200-1), it may request a first set of obfuscation arguments again from the data server (200-1).

[0157] As another example, when the processor (180) collects the original data of the second type and receives the first set of obfuscation arguments from the data server (200-1), it may request the second set of obfuscation arguments again from the data server (200-1).

[0158] The original data of Type 1 may be the number of views of a specific genre per time interval, and the original data of Type 2 may be the number of views of a specific channel per time interval, but this is merely an example.

[0159] The values ​​of the individual obfuscation arguments in the first obfuscation argument set and the second obfuscation argument set may differ from each other.

[0160] The processor (180) of the artificial intelligence device (100) can determine whether the degree of obfuscation included in the obfuscation factor data set is within a certain range (S703).

[0161] The degree of obfuscation may indicate the extent to which the obfuscated data reflects the original data. The processor (180) may determine that the degree of obfuscation is within a certain range if the value of the first obfuscation factor (ε) is greater than 0 and less than 0.1.

[0162] The value of the first obfuscation factor (ε) can be closer to 0.1, which increases the degree of obfuscation, and closer to 0, which decreases the degree of obfuscation.

[0163] If the value of the first obfuscation factor (ε) becomes too small, the obfuscated data may be difficult to reflect the trend of the original data. Accordingly, the processor (180) can determine whether the degree of obfuscation is within a certain range.

[0164] Figures 8a and 8b are figures comparing the obfuscation results of original data according to the degree of obfuscation.

[0165] Each original data point may represent the number of views of content videos of a specific genre by time period collected over several months. Fig. 8a is the case where the degree of obfuscation is 0.1, and Fig. 8b is the case where the degree of obfuscation is 0.086.

[0166] Referring to Fig. 8a, when the obfuscation level is 0.1, the obfuscated data follows the trend of the original data well. Referring to Fig. 8b, when the obfuscation level is 0.086, the obfuscated data does not follow the trend of the original data well. In other words, as the obfuscation level decreases, the obfuscated data may not reflect the characteristics of the original data.

[0167] Again, Fig. 7a is explained.

[0168] If the degree of obfuscation is not within a certain range, the processor (180) may request a new set of obfuscation arguments from the data server (200-1) (S705).

[0169] If the value of the first obfuscation factor (ε) is not within a certain range, the processor (180) may request the obfuscation factor set again from the data server (200-1) and receive the updated obfuscation factor set from the data server (200-1).

[0170] When the processor (180) of the artificial intelligence device (100) determines that the degree of obfuscation is within a certain range, it can perform a local differential privacy technique to generate obfuscated data (S707).

[0171] The processor (180) can generate obfuscated data from viewing history information based on the following [Equation 1].

[0172] [Mathematical Formula 1]

[0173]

[0174] Here, Pr represents the distribution M(D or D'), M is the difference mechanism, D is the comparison dataset (corresponding to viewing history information), and D' is the target dataset (corresponding to obfuscated data). The difference mechanism may follow either a Gaussian mechanism or a Laplace mechanism.

[0175] The comparison dataset and the target dataset may be described in discrete or continuous data forms depending on the number of contents included in the content list and the degree of notation by interval. Each of the comparison dataset and the target dataset may include a single figure, a time-series figure, or a cumulative figure per content.

[0176] The processor (180) of the artificial intelligence device (100) can determine whether there exists a value K representing the minimum number of labels for verifying the obfuscation result based on the generated obfuscation data (S709), and if there exists a value K, it can determine whether the obfuscation data includes the top K obfuscation data (S711).

[0177] K can be 3, but this is merely an example and may vary depending on the type of the source data. The value of K when the source data is the number of views of content of a specific genre by time period may differ from the value of K when the source data is the number of views of content containing a specific object by time period.

[0178] The processor (180) can extract the top K obfuscated data among the obfuscated data whose data values ​​are greater than or equal to a certain value. For example, if the original data is the number of views of a specific genre by time period, the processor (180) can determine whether the trend between the original data and the obfuscated data matches for the three time periods with the highest number of views.

[0179] The processor (180) can determine that the obfuscated data contains the top K obfuscated data if the trend between the original data and the obfuscated data matches for the three time periods with the highest number of views.

[0180] If verification is performed only on the top K obfuscated data, the speed of data analysis is increased, and statistical significance can be ensured.

[0181] The processor (180) of the artificial intelligence device (100) can determine whether the top K (Top-K) obfuscated data satisfy the significance range (S713).

[0182] The processor (180) can perform significance verification to ensure the reliability of the top K (Top-K) obfuscated data. The processor (180) can determine that the significance range is satisfied if the value of the second obfuscation factor (δ) for each of the top K (Top-K) obfuscated data is less than a threshold value.

[0183] The processor (180) may determine that the significance range is satisfied if the value obtained by subtracting the second obfuscation factor (δ) from 1 is greater than or equal to a set threshold value.

[0184] Step S713 may be a process of evaluating the difference in distribution between the original data and the original data.

[0185] The processor (180) of the artificial intelligence device (100) can store the obfuscated data in memory (170) and transmit the obfuscated data to the data server (200-1) when the top K obfuscated data satisfy the significance range (S715).

[0186] Referring to FIG. 7b, the processor (260) of the data server (200-1) can determine whether the representativeness of the obfuscated data received from the artificial intelligence device (100) is satisfied (S717).

[0187] The processor (260) of the data server (200-1) can determine that the representativeness of the obfuscated data is satisfied if the difference between the sample ratio of the obfuscated data and the sample ratio of the original data is less than a certain ratio.

[0188] The processor (260) of the data server (200-1) may determine that the representativeness of the obfuscated data is not satisfied if the difference between the sample ratio of the obfuscated data and the sample ratio of the original data is greater than a certain ratio.

[0189] When the representativeness of the obfuscated data received from the artificial intelligence device (100) is satisfied, the processor (260) of the data server (200-1) can obtain a viewing analysis result based on the obfuscated data according to S409 of FIG. 4, which will be described later (S409).

[0190] If the representativeness of the obfuscated data received from the artificial intelligence device (100) is not satisfied, the processor (260) of the data server (200-1) can update the set of obfuscated arguments (S719) and transmit the updated set of obfuscated arguments to the artificial intelligence device (100) (S721).

[0191] If the processor (260) of the data server (200-1) does not satisfy the representativeness of the obfuscated data received from the artificial intelligence device (100), it can send a new set of obfuscated arguments to the artificial intelligence device (100) to satisfy the representativeness.

[0192] Meanwhile, the processor (180) of the artificial intelligence device (100) can directly determine whether the obfuscation representativeness is satisfied. If the processor (180) determines that the representativeness of the obfuscated data is not satisfied, it can request a new set of obfuscation arguments from the data server (200-1) to satisfy the representativeness. The processor (180) can re-obfuscate the viewing history information based on the new set of obfuscation arguments and transmit the new obfuscated data resulting from the re-obfuscation to the data server (200-1).

[0193] Again, Figure 4 is explained.

[0194] The processor (180) of the artificial intelligence device (100) can transmit obfuscated data to the data server (200-1) through the communication interface (110) (S407).

[0195] The processor (260) of the data server (200-1) can obtain a viewing analysis result based on the received obfuscated data (S409).

[0196] The viewing analysis results may include at least one of the user's preferred genre, preferred channel, preferred person, or viewing pattern. Each of the preferred genre, preferred channel, and preferred person may be an example of a keyword.

[0197] The processor (180) can determine a specific user's viewing pattern by time period based on obfuscated data.

[0198] The processor (260) of the data server (200-1) can transmit the viewing analysis results to the operator server (300) through the communication interface (210) (S411).

[0199] The operator server (300) can obtain customized content information based on the received viewing analysis results (S413) and can transmit the obtained customized content information to the data server (200-1) (S415).

[0200] For example, personalized content information may include one or more of targeted advertisements or recommended content.

[0201] Targeted advertising can be advertisements that can be provided to specific users at specific times.

[0202] The processor (260) of the data server (200-1) can transmit customized content information received from the operator server (300) to the artificial intelligence device (100) through the communication interface (210) (S417). The data server (200-1) may also generate recommended content information based on the viewing analysis results and transmit the recommended content information along with the customized content information to the artificial intelligence device (100).

[0203] The processor (180) of the artificial intelligence device (100) can output customized content information through the output interface (150) (S419).

[0204] The processor (180) can display customized content information through the display (151). For example, the processor (80) can display a customized advertisement corresponding to a specific time period along with a content video on the display (151) during a portion of the specific time period.

[0205] The processor (180) can also display recommended content information on the display (151) that includes content of a genre similar to the genre preferred by the user at a specific time period.

[0206] In this way, according to an embodiment of the present disclosure, a personalized service can be provided to each of a plurality of users.

[0207] In addition, according to an embodiment of the present disclosure, the privacy of user data transmitted externally can be protected, thereby reducing damage from data leakage.

[0208] In addition, according to an embodiment of the present disclosure, the data server (200-1) can perform various statistical analyses based on parameter analysis even when receiving modified numerical data (obfuscated data).

[0209] In addition, according to an embodiment of the present disclosure, content can be recognized on a non-linear transmission platform, user interests can be derived to record viewing history, and proactive customer information can be built.

[0210] FIG. 9 is a diagram illustrating an example of providing a personalized service for each user according to an embodiment of the present disclosure.

[0211] The content provider (900) may include a broadcasting station, a content provider (910), and an advertiser (920). The data server (200-1) may receive obfuscated data corresponding to viewing history information from each of the multiple terminals (100-1, 100-2, 100-i) corresponding to each of the multiple users.

[0212] The data server (200-1) can obtain a viewing analysis result based on each obfuscated data and can receive targeted advertisements and recommended content from the content provider (900) based on the viewing analysis result.

[0213] The data server (200-1) can transmit different targeted advertisements and recommendation content to each of the multiple terminals (100-1, 100-2, 100-i).

[0214] FIG. 10 is a drawing for explaining the configuration of a data server according to one embodiment of the present disclosure.

[0215] Referring to FIG. 10, the data server (200-1) may include an obfuscated argument / content / keyword transmission unit (1010), an aggregate information receiving unit (1020), a storage unit (1030), an aggregate information transmission unit (1040), a content / keyword receiving unit (1050), and a traffic control unit (160).

[0216] The obfuscated argument / content / keyword transmission unit (1010), aggregate information reception unit (1020), aggregate information transmission unit (1040), and content / keyword reception unit (1050) may be included in the communication interface (210) of FIG. 2.

[0217] The storage unit (1030) may be included in the memory (230) of FIG. 2.

[0218] The traffic control unit (160) may be included in the processor (260) of FIG. 2.

[0219] The obfuscation argument / content / keyword transmission unit (1010) can transmit an obfuscation argument set, content, and keywords to an artificial intelligence device (100). Transmission of a third obfuscation argument of the obfuscation argument set may be optional.

[0220] The aggregate information receiving unit (1020) can receive obfuscated data from the artificial intelligence device (100). The received obfuscated data can be stored in the aggregate information area of ​​the storage unit (1030).

[0221] The aggregate information transmission unit (1040) can transmit accumulated data recorded during the transmission and reception process with the artificial intelligence device (100) to the operator server (300). The aggregate information transmission unit (1040) can transmit the viewing analysis results to the operator server (300).

[0222] The content / keyword receiving unit (1050) can receive identifiers, content data, or keyword data of the content to be transmitted requested by the operator server (300). The received data can be recorded in each storage area of ​​the storage unit (1030).

[0223] FIG. 11 is a drawing for explaining the configuration of a data server according to one embodiment of the present disclosure.

[0224] Referring to FIG. 10, the data server (200-1) may include an aggregate information request unit (310), an aggregate information receiving unit (320), a content and keyword lookup unit (330), and a content and keyword transmission unit (340).

[0225] The aggregate information request unit (310) can request a viewing analysis result corresponding to the obfuscated data from the data server (200-1). The aggregate information request unit (310) can request a viewing analysis result corresponding to a specific type of viewing history information from the data server (200-1).

[0226] The aggregate information request unit (310) may also request the viewing analysis results for the identification device desired by the business operator from the data server (200-1).

[0227] The aggregate information receiving unit (320) can receive a viewing analysis result from the data server (200-1) in response to a request from the aggregate information requesting unit (310).

[0228] The content and keyword lookup unit (330) can search for customized content using one or more of the keywords, the name of the content, the genre, or the type included in the viewing analysis results. The keyword may be any one of the subject of the content, part of the name of the content, or the name of an object included in the content.

[0229] The content and keyword transmission unit (340) can transmit customized content information for the searched customized content to the data server (200-1).

[0230] An artificial intelligence device (100) according to one embodiment of the present disclosure may include: a communication interface (110) that communicates with a display (151) and a server (200-1); a memory (170) that stores an artificial intelligence model that infers a content recognition result based on one or more of video, audio, or text output from the display; and a processor (180) that acquires viewing history information including the content recognition result, generates obfuscated data from the viewing history information using an obfuscation technique, and transmits the generated obfuscated data to the server through the communication interface.

[0231] The set of obfuscation arguments used in the above obfuscation technique may vary depending on the type of the above viewing history information.

[0232] The above obfuscation technique may be a Local Differential Privacy technique, and the set of obfuscation factors may include a first obfuscation factor (ε) representing the degree of obfuscation, a second obfuscation factor (δ) representing the statistical significance range, and a third obfuscation factor (K) representing the minimum number of labels for verifying the obfuscation result.

[0233] The above content recognition result may include at least one of the name of an object included in the scene of the content video, the situation of the scene, the content of the scene, the genre of the content video, or the name of the content video.

[0234] In addition to the content recognition results, the above viewing history information may further include at least one of the following: the genre of the content video watched by the user by time period, the number of times a specific genre is watched by time period, the viewing time or valid viewing interval of each content video, or a summary of the content of the content video.

[0235] The processor (180) may request a new set of obfuscation arguments from the server based on the fact that the representativeness of the obfuscated data is not satisfied, receive the new set of obfuscation arguments in accordance with the request, and generate new obfuscated data from the viewing history information based on the new set of obfuscation arguments.

[0236] The processor (180) can receive custom content information in response to the transmission of the obfuscated data from the server (200-1) via the communication interface (110), and the custom content information may include at least one of a targeted advertisement or recommended content.

[0237] The processor (180) can receive new labeling data for learning the artificial intelligence model from the server (200-1) through the communication interface (110) and retrain the artificial intelligence model.

[0238] The above-described disclosure can be implemented as computer-readable code on a medium on which a program is recorded. A computer-readable medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include a Hard Disk Drive (HDD), a Solid State Disk (SSD), a Silicon Disk Drive (SDD), ROM, RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc. Additionally, the computer may include a processor (180) of an artificial intelligence device.

Claims

1. Regarding artificial intelligence devices, display; A communication interface that communicates with a server; A memory storing an artificial intelligence model that infers a content recognition result based on one or more of video, audio, or text output from the above display; and A processor comprising: acquiring viewing history information including the content recognition result; generating obfuscated data from the viewing history information using an obfuscation technique; and transmitting the generated obfuscated data to the server through the communication interface. Artificial intelligence device.

2. In Paragraph 1, The set of obfuscation arguments used in the above obfuscation technique varies depending on the type of the above viewing history information. Artificial intelligence device.

3. In Paragraph 2, The above obfuscation technique is It is a Local Differential Privacy technique, and The above set of obfuscated arguments is A first obfuscation factor (ε) indicating the degree of obfuscation, a second obfuscation factor (δ) indicating the statistical significance range, and a third obfuscation factor (K) indicating the minimum number of labels for verifying the obfuscation result. Artificial intelligence device.

4. In Paragraph 1, The above content recognition result is Includes at least one of the name of an object included in a scene of the content video, the situation of the scene, the content of the scene, the genre of the content video, or the name of the content video. Artificial intelligence device.

5. In Paragraph 4, The above viewing history information In addition to the above content recognition results, at least one of the following is further included: the genre of content videos watched by the user by time period, the number of times a specific genre is watched by time period, the viewing time or valid viewing interval of each content video, or a summary of the content of the content video. Artificial intelligence device.

6. In Paragraph 2, The above processor Based on the fact that the representativeness of the above obfuscated data is not satisfied, a new set of obfuscation arguments is requested from the server, a new set of obfuscation arguments is received in accordance with the request, and new obfuscated data is generated from the viewing history information based on the new set of obfuscation arguments. Artificial intelligence device.

7. In Paragraph 1, The above processor Receiving customized content information in response to the transmission of the obfuscated data from the server through the communication interface, The above customized content information is including at least one of targeted advertisements or recommended content Artificial intelligence device.

8. In Paragraph 1, The above processor Receiving new labeling data for training the artificial intelligence model from the server through the communication interface and retraining the artificial intelligence model Artificial intelligence device.

9. In the method of operating an artificial intelligence device, A step of outputting one or more of video, audio, or text; A step of inferring content recognition results based on one or more of video, audio, or text output through an artificial intelligence model; A step of obtaining viewing history information including the above-mentioned content recognition result; A step of generating obfuscated data from the viewing history information using an obfuscation technique; and Includes the step of transmitting the generated obfuscated data to a server Method of operation of an artificial intelligence device.

10. In Paragraph 1, The set of obfuscation arguments used in the above obfuscation technique varies depending on the type of the above viewing history information. Method of operation of an artificial intelligence device.

11. In Paragraph 2, The above obfuscation technique is It is a Local Differential Privacy technique, and The above set of obfuscated arguments is A first obfuscation factor (ε) indicating the degree of obfuscation, a second obfuscation factor (δ) indicating the statistical significance range, and a third obfuscation factor (K) indicating the minimum number of labels for verifying the obfuscation result. Method of operation of an artificial intelligence device.

12. In Paragraph 1, The above content recognition result is Includes at least one of the name of an object included in a scene of the content video, the situation of the scene, the content of the scene, the genre of the content video, or the name of the content video. Method of operation of an artificial intelligence device.

13. In Paragraph 12, The above viewing history information In addition to the above content recognition results, at least one of the following is further included: the genre of content videos watched by the user by time period, the number of times a specific genre is watched by time period, the viewing time or valid viewing interval of each content video, or a summary of the content of the content video. Method of operation of an artificial intelligence device.

14. In Paragraph 10, A step of requesting a new set of obfuscation arguments from the server based on the fact that the representativeness of the obfuscated data is not satisfied; The step of receiving the new set of obfuscated arguments in accordance with the above request; and The method further includes the step of generating new obfuscated data from the viewing history information based on the new set of obfuscated arguments. Method of operation of an artificial intelligence device.

15. In Paragraph 9, The method further includes the step of receiving customized content information in response to the transmission of the obfuscated data. The above customized content information is including at least one of targeted advertisements or recommended content Method of operation of an artificial intelligence device.

Images