Method, computer program, and device for preventing unauthorized photographing of display screen

Abstract

A method for preventing unauthorized photographing of a display screen, which is performed by a computing device including at least one processor, according to an embodiment of the present disclosure includes the steps of: setting an operation level by measuring hardware and software performance of the computing device; generating image data by photographing a space adjacent to the front of the display screen; and inputting the image data and the operation level to an artificial neural network-based detection model and predicting a dangerous event.

Description

Method, computer program, and device for preventing unauthorized filming of a display screen

[0001] The content of the present disclosure relates to a method for preventing unauthorized filming of a display screen, and more specifically, to a method, computer program, and device for preventing filming of security data displayed on a display screen.

[0002] As semiconductor circuits become smaller and finer, and high-density integrated circuit technologies advance, camera modules capable of capturing objects are being installed even in small smartphones, replacing conventional cameras. Individuals can easily photograph desired objects while carrying small smartphones for personal use. Furthermore, it has recently become common practice to handle work within a company using ICT technology. Typically, employees working in-house acquire and process information using display screens on monitors such as desktops or laptops. In this scenario, individuals can easily access security data displayed on the screen that is prohibited from being leaked externally, and they can easily photograph and possess such data. Moreover, not only are there frequent instances where individuals intentionally photograph and leak security data, but also cases where security data in their possession is hacked and leaked externally. While identifying markers such as watermarks are displayed on screens to prevent such data leakage, this only allows for the identification of the source of the leak and cannot proactively block the leakage of security data.

[0003] The present disclosure is devised in response to the aforementioned background technology and aims to provide a method, computer program, and device for preventing the photographing of security data displayed on a display screen.

[0004] In addition, the present disclosure aims to provide a method for recognizing a risk situation and performing a security action by selecting one or more of a plurality of detection models according to the performance of a computing device.

[0005] However, the problems to be solved in this disclosure are not limited to those described above, and problems not mentioned will be clearly understood by those skilled in the art to which this disclosure belongs from this specification and the attached drawings.

[0006] A method for preventing unauthorized filming of a display screen, performed by a computing device comprising at least one processor according to one embodiment of the present disclosure, comprises the steps of: measuring the hardware and software performance of the computing device to set an operation level; filming a space adjacent to the front of the display screen to generate image data; and inputting the image data and the operation level into an artificial neural network-based detection model to predict a risk event.

[0007] Alternatively, the above-mentioned operating level is characterized by being set based on the type of processor included in the computing device or the operating speed of the artificial neural network-based sensing model.

[0008] Alternatively, the artificial neural network-based sensing model includes a plurality of sub-sensing models and is characterized by selecting and operating one or more of the plurality of sub-sensing models according to the operation level.

[0009] Alternatively, the artificial neural network-based detection model is characterized by including one or more of a face detection model, a face feature point and expression detection model, an object classification model, and a human body pose estimation model.

[0010] Alternatively, the artificial neural network-based detection model is characterized by predicting a risk event based on elbow angle information predicted by the human body posture estimation model.

[0011] Alternatively, the above-mentioned operation level is characterized by being reset during the operation of the computing device.

[0012] Alternatively, it includes a step of performing a security action, including logging, according to the above-mentioned operation level and detected risk event.

[0013] Alternatively, the log record is characterized by including one or more of the following: operation information of the computing device at the time the risk event is detected, the image data, the computing device information, user information, the type of the detected risk event, and the event detection time information.

[0014] Alternatively, the security operation is characterized by including one or more of log recording, displaying a watermark on the display screen, and displaying a security pattern on the display screen.

[0015] Alternatively, the watermark or security pattern is characterized by being displayed only on a portion of the frame of the display screen, so that it is displayed only on the captured image among the image of the display screen and the eyes of the person viewing the display screen.

[0016] A computer program stored on a computer-readable storage medium according to one embodiment of the present disclosure, wherein the computer program performs operations to prevent unauthorized photographing of a display screen when executed on one or more processors. The operations include an operation of setting an operation level by measuring the hardware and software performance of the computing device, an operation of generating image data by photographing a space adjacent to the front of the display screen, and an operation of inputting the image data and the operation level into an artificial neural network-based detection model and predicting a risk event. The operation level is set based on the type of processor included in the computing device or the operation speed of the artificial neural network-based detection model, and the artificial neural network-based detection model includes a plurality of sub-detection models, and is characterized by selecting and operating one or more of the plurality of sub-detection models according to the operation level.

[0017] A computing device for preventing unauthorized filming of a display screen according to one embodiment of the present disclosure includes a processor comprising at least one core, a memory comprising program codes executable on the processor, and a network unit. The processor measures the hardware and software performance of the computing device to set an operation level, films a space adjacent to the front of the display screen to generate image data, inputs the image data and the operation level into an artificial neural network-based detection model to predict a risk event, and performs security operations including log recording according to the operation level and the detected risk event. The operation level is set based on the type of processor included in the computing device or the operation speed of the artificial neural network-based detection model, and the artificial neural network-based detection model includes a plurality of sub-detection models, and is characterized by selecting and operating one or more of the plurality of sub-detection models according to the operation level.

[0018] According to one embodiment of the present disclosure, a risk event, such as capturing security material displayed on a display, can be detected and a security action performed on a computing device of various performance capabilities.

[0019] According to one embodiment of the present disclosure, even when the performance of the user device on which the on-device AI model operates varies, the AI ​​model can be flexibly selected to match the performance of the user device.

[0020] Through the present disclosure, when a computing device is of low specifications, the minimum functions of a security program can be reliably executed, and when a high-specification device is used, advanced functions can be utilized, thereby minimizing the impact on other tasks when the security program is running.

[0021] In addition, since the present disclosure integrates various technologies such as face recognition, object recognition, and pose estimation, it can be utilized in various application fields, including not only security systems but also user authentication, behavior analysis, and smart environment control.

[0022] The effects of the present disclosure are not limited to the effects described above, and unmentioned effects will be clearly understood by those skilled in the art from the present specification and the accompanying drawings.

[0023] FIG. 1 is a block diagram of a computing device according to one embodiment of the present disclosure.

[0024] FIG. 2 is a block diagram schematically showing a display unauthorized shooting prevention system according to one embodiment of the present disclosure.

[0025] FIG. 3 is a drawing illustrating a display unauthorized shooting prevention device according to one embodiment of the present disclosure.

[0026] FIG. 4 is a diagram illustrating the operation of a sensing model according to one embodiment.

[0027] FIG. 5 is a flowchart illustrating a method for preventing unauthorized filming of a display screen according to one embodiment.

[0028] FIG. 6 is a flowchart illustrating a method for preventing unauthorized filming of a display screen according to one embodiment.

[0029] Embodiments of the present disclosure are described below with reference to the attached drawings so that those skilled in the art (hereinafter, those skilled in the art) can easily implement them. The embodiments presented in the present disclosure are provided to enable those skilled in the art to use or implement the contents of the present disclosure. Accordingly, various modifications to the embodiments of the present disclosure will be apparent to those skilled in the art. That is, the present disclosure may be embodied in various different forms and is not limited to the embodiments below.

[0030] Throughout the specification of the present disclosure, identical or similar reference numerals refer to identical or similar components. Additionally, to clearly explain the present disclosure, reference numerals in the drawings that are unrelated to the description of the present disclosure may be omitted.

[0031] The term “or” as used in this disclosure is intended to mean an implicit “or” rather than an exclusive “or.” That is, unless otherwise specified in this disclosure or its meaning is not clear from the context, “X uses A or B” should be understood to mean one of the natural implicit substitutions. For example, unless otherwise specified in this disclosure or its meaning is not clear from the context, “X uses A or B” may be interpreted as X using A, X using B, or X using both A and B.

[0032] The term “and / or” as used in this disclosure should be understood to refer to and include all possible combinations of one or more of the enumerated related concepts.

[0033] The terms “comprising” and / or “comprising” as used in this disclosure should be understood to mean the presence of certain features and / or components. However, the terms “comprising” and / or “comprising” should be understood not to exclude the presence or addition of one or more other features, other components and / or combinations thereof.

[0034] Where not otherwise specified in the present disclosure or where it is not clear from the context that the singular form indicates, the singular should generally be interpreted as including “one or more.”

[0035] The term “the N (N is a natural number)” used in this disclosure may be understood as an expression used to distinguish the components of this disclosure from one another according to certain criteria, such as functional perspectives, structural perspectives, or convenience of explanation. For example, components performing different functional roles in this disclosure may be distinguished as a first component or a second component. However, components that are substantially identical within the technical scope of this disclosure but must be distinguished for the convenience of explanation may also be distinguished as a first component or a second component.

[0036] The term “acquisition” as used in this disclosure can be understood to mean not only receiving data through a wired or wireless communication network with an external device or system, but also generating data in an on-device form.

[0037] Meanwhile, the terms "module" or "unit" used in this disclosure may be understood as referring to an independent functional unit that processes computing resources, such as a computer-related entity, firmware, software or a part thereof, hardware or a part thereof, or a combination of software and hardware. In this case, "module" or "unit" may be a unit composed of a single element, or a unit expressed as a combination or set of multiple elements. For example, in a narrow sense, "module" or "unit" may refer to a hardware element of a computing device or a set thereof, an application program that performs a specific function of software, a procedure implemented through software execution, or a set of instructions for program execution. Furthermore, in a broad sense, "module" or "unit" may refer to the computing device itself that constitutes the system, or an application executed on the computing device. However, since the above-described concept is merely an example, the concepts of "module" or "part" may be defined in various ways within the scope understandable to those skilled in the art based on the contents of this disclosure.

[0038] As used in this disclosure, the term "model" may be understood as a system implemented using mathematical concepts and language to solve a specific problem, a set of software units to solve a specific problem, or an abstract model regarding a processing process to solve a specific problem. For example, a neural network "model" may refer to an overall system implemented as a neural network that possesses problem-solving capabilities through learning. In this case, the neural network may possess problem-solving capabilities by optimizing parameters connecting nodes or neurons through learning. A neural network "model" may include a single neural network or a set of neural networks composed of multiple neural networks.

[0039] The term “data” as used in this disclosure may include “images,” signals, etc. The term “image” as used in this disclosure may refer to multidimensional data composed of discrete image elements. In other words, “image” may be understood as a term referring to a digital representation of an object visible to the human eye. For example, “image” may refer to multidimensional data composed of elements corresponding to pixels in a two-dimensional image.

[0040] The term "inductive bias" as used in this disclosure can be understood as a set of assumptions that enable inductive reasoning of a deep learning model. In order to resolve the error of generalization, where a deep learning model performs adequately only on given training data, it is necessary for the deep learning model to reason so that it approaches an accurate output even on data other than the given training data. Therefore, "inductive bias" can be understood as a set of preconditions that a deep learning model has during the learning process to predict the output of an ungiven input.

[0041] The term "block" as used in this disclosure can be understood as a set of configurations classified based on various criteria such as type, function, etc. Accordingly, configurations classified as a single "block" may be varied depending on the criteria. For example, a neural network "block" can be understood as a set of neural networks including at least one neural network. In this case, it can be assumed that the neural networks included in the neural network "block" perform specific operations identically.

[0042] The term "operation function" as used in this disclosure can be understood as a mathematical expression for a constituent unit that performs a specific function or processes an operation. For example, the "operation function" of a neural network block can be understood as a mathematical expression representing a neural network block that processes a specific operation. Accordingly, the relationship between the input and output of a neural network block can be expressed as a formula through the "operation function" of the neural network block.

[0043] The explanation of the foregoing terms is intended to aid in understanding the present disclosure. Accordingly, it should be noted that unless a foregoing term is explicitly stated as a matter limiting the content of the present disclosure, it is not to be used in the sense of limiting the technical concept of the content of the present disclosure.

[0044] FIG. 1 is a block diagram of a computing device according to one embodiment of the present disclosure.

[0045] A computing device (100) according to one embodiment of the present disclosure may be a hardware device or part of a hardware device that performs comprehensive processing and computation of data, or it may be a software-based computing environment connected to a communication network. For example, the computing device (100) may be a server that performs intensive data processing functions and is an entity that shares resources, or it may be a client that shares resources through interaction with the server. Additionally, the computing device (100) may be a cloud system in which a plurality of servers and clients interact to comprehensively process data. Since the above description is merely one example regarding the type of computing device (100), the type of computing device (100) may be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

[0046] Referring to FIG. 1, a computing device (100) according to one embodiment of the present disclosure may include a processor (110), a memory (120), and a network unit (130). However, since FIG. 1 is merely an example, the computing device (100) may include other configurations for implementing a computing environment. Additionally, only some of the disclosed configurations may be included in the computing device (100).

[0047] A processor (110) according to one embodiment of the present disclosure may be understood as a constituent unit comprising hardware and / or software for performing computing operations. For example, the processor (110) may read a computer program and perform data processing for machine learning. The processor (110) may process computational processes such as processing input data for machine learning, extracting features for machine learning, and calculating errors based on backpropagation. A processor (110) for performing such data processing may include a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA). Since the above-described type of processor (110) is merely an example, the type of processor (110) may be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

[0048] The processor (110) can monitor a specific area and predict security-related risk events by using multiple neural network models, such as a neural network model for detecting objects, a neural network model for detecting faces, a neural network model for detecting facial feature points and expressions, a neural network model for classifying objects, and a neural network model for estimating poses. For example, neural network models such as MediaPipe’s EfficientDet-Lite2, BlazeFace, FaceLandmarker, BlazePose, or YOLO (You Only Look Once) can be utilized. The processor (110) can train the neural network model using an image captured by a camera as input to predict behaviors that violate security policies, such as a user attempting illegal filming.

[0049] The processor (110) can predict behaviors that violate the security policy by using a neural network model generated through the learning process described above. In addition to the example described above, behaviors that violate the security policy can be predicted by utilizing various data from a computing device as input, in addition to images captured by a camera, and the inputs and outputs of the neural network model can be configured in various ways within a scope understandable to those skilled in the art based on the content of the present disclosure.

[0050] A memory (120) according to one embodiment of the present disclosure may be understood as a configuration unit comprising hardware and / or software for storing and managing data processed by a computing device (100). That is, the memory (120) may store data of any form generated or determined by a processor (110) and data of any form received by a network unit (130). For example, the memory (120) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory, RAM (random access memory), SRAM (static random access memory), ROM (read-only memory), EEPROM (electrically erasable programmable read-only memory), PROM (programmable read-only memory), magnetic memory, a magnetic disk, and an optical disk. Additionally, the memory (120) may include a database system that controls and manages data in a predetermined system. Since the above-described type of memory (120) is merely an example, the type of memory (120) can be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

[0051] The memory (120) can structure and organize data, combinations of data, and program code executable by the processor (110) that are necessary for the processor (110) to perform calculations. For example, the memory (120) can store image data received through a camera module to be described later. The memory (120) can store program code that causes a neural network model to receive image data and perform learning, program code that causes a neural network model to receive image data and perform inference according to the purpose of use of the computing device (100), and processed data generated as the program code is executed.

[0052] A network unit (130) according to one embodiment of the present disclosure may be understood as a configuration unit that transmits and receives data through any known form of wired or wireless communication system. For example, the network unit (130) may perform data transmission and reception using wired or wireless communication systems such as a local area network (LAN), wideband code division multiple access (WCDMA), long term evolution (LTE), wireless broadband internet (WiBro), 5th generation mobile communication (5G), ultra-wide-band wireless communication, ZigBee, radio frequency (RF) communication, wireless LAN, wireless fidelity (Wi-Fi), near field communication (NFC), or Bluetooth. Since the communication systems described above are merely examples, wired or wireless communication systems for data transmission and reception of the network unit (130) may be applied in various ways other than those described above.

[0053] The network unit (130) can receive data necessary for the processor (110) to perform calculations through wired or wireless communication with any system or any client. Additionally, the network unit (130) can transmit data generated through the calculations of the processor (110) through wired or wireless communication with any system or any client. For example, the network unit (130) can transmit log records regarding risk events that violate security policies through communication with a cloud server or computing device. The network unit (130) can transmit and receive intermediate data, processed data, etc. related to security operations through communication with the aforementioned database, server, or computing device.

[0054] FIG. 2 is a block diagram schematically showing a display unauthorized shooting prevention system according to one embodiment of the present disclosure.

[0055] Referring to FIG. 2, a display unauthorized shooting prevention system may include a computing device (100), a display (210), a camera module (220), and a server (300) connected to the computing device (100) via a network. The computing device (100) may include a performance measurement module (110), an event detection module (120), and a security module (130). The computing device (100) may further include various configurations such as a communication module, an input module, a screen saver module, and a log recording module to prevent unauthorized shooting of the display.

[0056] A display (210) according to one embodiment displays a screen. Additionally, the display (210) may display a screen saver and a security pattern, as described below, on the screen. The display (210) according to one embodiment should be understood as encompassing desktop monitors, mobile devices (a concept including PDPs, mobile phones, etc.), portable laptop monitors, etc. Additionally, the types of display screens include all known types of display screens, such as LCDs (Liquid Crystal Displays), PDPs (Plasma Display Panels), and OLEDs (Organic Light Emitting Diodes).

[0057] The camera module (220) can capture and record images of the front of the display (210) screen. That is, the camera module (220) may be a video device consisting of a lens and a recorder. For example, the camera module (220) may be any one of the known web cams. The camera module (220) may be provided on the display (210).

[0058] A log server (300) according to one embodiment is connected wirelessly or via a wired connection to a system (including a security module (130)) such as a computing device (100) having a display (210), and can receive and record log data regarding events, activities, and image data captured at the corresponding time occurring in the system or OS (SW).

[0059] The performance measurement module (110) can measure the hardware and software performance of the computing device (100). The performance measurement module (110) can determine whether the computing device (100) includes a graphics processing unit (GPU) and set an operation level. The event detection module (120) can operate an artificial neural network-based detection model configured to operate on the GPU or CPU depending on the operation level.

[0060] The performance measurement module (110) can select a model that is operable on the computing device (100) from among a plurality of sub-sensing models included in the artificial neural network-based sensing model by comparing the measured performance with a preset standard. The performance measurement module (110) can measure the operation speed or response speed for each of the plurality of sub-sensing models, and can measure the operation speed or response speed for a combination including one or more of the plurality of sub-sensing models. The operation level may include the performance of the computing device (100), information on the sub-sensing models operable on the computing device (100), and information on the operations to be performed by the security module (130) after the event detection module (120) detects an event.

[0061] The event detection module (120) can detect events by receiving image data captured by the camera module (220). For example, the event detection module (120) can receive video data regarding the frontal area of ​​the display (210) screen captured and recorded from the camera module (220), and predict events related to unauthorized filming of the display (210) screen or leakage of security materials displayed on the display (210) screen from the video data. The event detection module (120) may include an artificial neural network model for predicting events. The event detection module (120) may include a plurality of neural network models, such as a neural network model for detecting objects, a neural network model for detecting faces, a neural network model for detecting facial feature points and expressions, a neural network model for classifying objects, and a neural network model for estimating poses.

[0062] The event detection module (120) can select a neural network model to perform event detection based on the performance or operation level measured by the performance measurement module (110).

[0063] The security module (130) may include a log management unit, a screen protection (screen saver) unit, and a security pattern generation unit.

[0064] When a preset event occurs, the log management unit can transmit an event log to the server (300) including the screen currently in operation, image data captured by the camera module (220), device ID, user ID, timestamp, and event type (reason). In an additional embodiment, the log management unit can transmit the event log to the server (300) based on a preset period or a program or task executed on the computing device (100).

[0065] A screen saver unit can disable the screen to prevent information contained on it from leaking. A screen saver unit can hide or protect sensitive information displayed on the screen when a pre-configured event occurs or while the user is away. A screen saver unit can be unlocked via a password or biometric authentication (e.g., facial recognition) in combination with a screen lock function.

[0066] The security pattern generation unit can display a specific pattern on all or part of the screen when a preset event occurs, and can display a specific pattern on some frames of the entire frame. The security pattern may include a watermark containing user information.

[0067] The security pattern may be displayed on the remaining frames, excluding at least some of the total frames displayed on the display (210) screen. In some frames, work images, such as security materials for a worker to perform work, are normally displayed on the display (210) screen.

[0068] Here, the term "total frame" implies that, for example, if the display (210) has a screen refresh rate of 60 Hz, the total frame displayed on the exemplary display (210) screen may be 60 frames. Additionally, "partial frame" may refer to a frame that can be perceived by a worker. Since a typical person (worker) cannot perceive anything faster than approximately 22 Hz due to the difference in speed between the organ that receives light and the organ that transmits it, in the exemplary embodiment, the partial frame may be selected based on 22 Hz. In this case, the security pattern may not be perceived by the eyes of a person viewing the display (210) screen, but may be displayed on an image captured of the display (210). Furthermore, it is desirable that the partial frame in which the work image is displayed and the remaining frame in which the security pattern is displayed alternately are displayed on the screen of the display (210). This is because if the frames in which the work image is displayed are concentrated in one section of the total frame and the frames in which the security pattern is displayed are concentrated in another section, it becomes difficult for the worker to perceive the work image, which may reduce the worker's work efficiency.

[0069]

[0070] FIG. 3 is a drawing illustrating a display unauthorized shooting prevention device according to one embodiment of the present disclosure.

[0071] Referring to FIGS. 2 and FIGS. 3, a camera module (220) may be positioned above the display (210) to capture and record an image of the front (d1) of the display (210) screen. The camera module (220) may be included in the display (210). The configuration of the camera module (220) and the display (210) in FIG. 3 is an exemplary form for illustrative purposes and may include various forms for capturing the front of the display (210) screen.

[0072] For example, as illustrated in FIG. 3, when a user (410) attempts to photograph the display (210) screen using a mobile phone (420), the computing device (100) can detect this. The computing device (100) can receive an image captured by the camera module (220) and recognize the user (410) and the mobile phone (420). The computing device (100) can recognize that the mobile phone (420) is positioned in the user's (410) hand and can predict that the user (410) is attempting to photograph the display (210) with the mobile phone (420) by analyzing the user's (410) posture.

[0073]

[0074] FIG. 4 is a diagram illustrating the operation of a sensing model according to one embodiment.

[0075] Referring to FIG. 4, the artificial neural network-based detection model included in the event detection module can predict a person, face, facial expression, or posture using image data as input. For example, the detection model can recognize a person (400) located in front of a display or in front of a camera. The detection model can extract feature points (0 to 31) of the recognized person (400) and determine the posture or body parts based on the extracted feature points. Based on the extracted feature points, the detection model can determine the position of the eyes, the direction the head is facing, the degree of tilt, closed eyes, distance from the lips, etc.

[0076] The detection model determines the position (13) of the elbow and the angle (11, 13, 15) at which the elbow is bent, and can adjust the weight when detecting an event based on the elbow angle. For example, if the detection model determines that the elbow angle is between 80 and 120 degrees, it can assign a weight of 3, indicating that there is a very high probability that the action is holding the phone and taking a picture of the display. If the elbow angle is between 50 and 80 degrees or between 120 and 150 degrees, the detection model can assign a weight of 2, indicating that there is a high probability that the action is holding the phone and taking a picture of the display.

[0077]

[0078] FIG. 5 is a flowchart illustrating a method for preventing unauthorized filming of a display screen according to one embodiment.

[0079] Referring to FIG. 5, the computing device can run a screen saver (S110). The computing device can store the screen being worked on, image data captured by the camera module, device ID, user ID, timestamp, event type (reason), etc., as a log before and after running or ending the screen saver.

[0080] The computing device may wait for the input of image data captured by the camera module before terminating the screen saver operation (S115). For example, the computing device may input image data when a face is recognized in the basic face recognition model that operates initially.

[0081] The computing device can measure the environment, hardware, or software performance in which a program is operated to prevent unauthorized filming of the display screen. The computing device can set an operation level based on the type of processor and the operation speed of the measured sub-detection models, select a detection model, and operate the selected detection model (S120). The computing device can execute all or part of a face detection model, a phone or camera detection model, a pose estimation model, and a face feature point extraction model using the image data recognized in S115 as input.

[0082] The computing device can perform face detection (S210). The computing device can calculate a threshold value based on input image data (S220) and determine whether the threshold is exceeded (S230). If the threshold is exceeded, the computing device can activate a screen saver (S140), and if the threshold is not exceeded, it can return to the work screen (S130).

[0083] The computing device can perform phone or camera detection (S310). If the phone or camera is not detected, the computing device can return to the work screen (S130). The computing device can calculate a threshold value using image data as input and determine whether the phone or camera is detected by comparing it with a preset threshold value (S315). If the phone or camera is detected, the computing device determines whether the pose estimation model is operating (S330), and if the pose estimation model is not operating, it can calculate a threshold value (S340). Depending on whether the threshold value is exceeded, the computing device can activate a screen saver or return to the work screen.

[0084] The computing device can perform pose estimation (S410). The computing device can select landmarks and calculate 3D coordinates (S415). The computing device can extract a risk area from the estimated pose (S420). The computing device can determine the risk of the estimated pose and the extracted area and adjust the weight (S440). For example, the computing device can determine the risk based on the degree to which the elbow is bent and use the risk as a weight when a phone or camera detection action occurs.

[0085] The computing device can extract facial feature points (S510). The computing device can interpret an action based on the extracted feature points (S520) and calculate a 3D coordinate threshold (S530). Based on the extracted feature points, the computing device can predict the direction the face is facing, the degree to which the head is lowered, the distance from the display screen, etc.

[0086] FIG. 5 is an exemplary embodiment for explaining a method of operating some of N sensing models according to the performance of a computing device, and the types and number of sensing models can be combined in various forms.

[0087]

[0088] FIG. 6 is a flowchart illustrating a method for preventing unauthorized filming of a display screen according to one embodiment.

[0089] Referring to FIG. 6, the computing device can measure device performance (S610). The computing device can measure the type of processor included in the computing device or the operating speed of the artificial neural network-based sensing model.

[0090] The computing device can set an operation level (S620). The operation level can be set based on the type of processor included in the computing device or the operation speed of the artificial neural network-based detection model. The computing device can set whether to operate or the operation level of the detection model by measuring the time it takes for the artificial neural network-based detection model to detect an event using image data as input and comparing it with a preset reference value.

[0091] The operation level can be reset during the operation of the computing device. The computing device can measure device performance and set the operation level at the time when a user logs into the computing device, or at the time when a specific program for a task or a pre-configured program is executed.

[0092] The computing device can select one or more sub-sensing models among a plurality of sub-sensing models included in an artificial neural network-based sensing model according to a set operation level.

[0093] An artificial neural network-based detection model may include one or more of a face detection model, a face feature point and expression detection model, an object classification model, and a human body pose estimation model.

[0094] The computing device can predict a danger event based on captured image data (S630). For example, the computing device can predict a danger event based on elbow angle information predicted by an artificial neural network-based detection model or a human body posture estimation model.

[0095] The computing device can perform security operations (S640). The computing device can perform security operations, including logging, depending on the operation level and detected risk events.

[0096] The log record may include one or more of the following: operation information of the computing device at the time the risk event is detected, image data, computing device information, user information, type of detected risk event, and event detection time information. The security action may include one or more of the following: log record, displaying a watermark on the display screen, and displaying a security pattern on the display screen. The watermark or security pattern may be displayed only on a portion of the frame of the display screen so that it is displayed only on the captured image, between the image capturing the display screen and the eyes of the person viewing the display screen.

[0097]

[0098] For example, a computing device can black out the screen when a risk event is detected. In this case, the taskbar at the bottom of the screen may not be obscured. The blacked-out screen may include UI elements, such as a button to escape the blackout state. The computing device can record logs regarding the basis for the high-risk assessment and the process of escaping the high-risk status. For instance, if the blackout state is resolved by pressing a button, a record such as "User requested screen resume" may be recorded.

[0099] The computing device can display a masking pattern and a watermark on the screen when a dangerous event is detected. When a dangerous event is detected, the computing device can leave only a log record on the display screen without separate intervention.

[0100] The computing device can transmit data related to risk events to the log server. The log server can subdivide the grade of the received risk events. When the log server receives a pre-configured risk event, it can notify the administrator by generating an alarm or similar mechanism. For example, the log server can record user IDs, timestamps, webcam images, screen images, security-related information, connected external display information, leakage factors, and assessed leakage risk levels.

[0101] The above detailed description is illustrative of the present disclosure. Furthermore, the foregoing describes preferred embodiments of the present disclosure, and the present disclosure may be used in various other combinations, modifications, and environments. That is, modifications or alterations are possible within the scope of the concept of the present disclosure, the scope equivalent to the described disclosure, and / or the scope of the art or knowledge. The described embodiments describe the best state for implementing the technical concept of the present disclosure, and various modifications required for specific fields of application and uses of the present disclosure are possible. Accordingly, the above detailed description of the invention is not intended to limit the present disclosure to the disclosed embodiments. Furthermore, the appended claims should be interpreted as including other embodiments.

Claims

1. A method for preventing unauthorized filming of a display screen, performed by a computing device comprising at least one processor, wherein A step of measuring the hardware and software performance of the computing device and setting an operation level; A step of generating image data by capturing a space adjacent to the front of the display screen; and The step of inputting the image data and the operation level into an artificial neural network-based detection model and predicting a risk event; comprising method.

2. In Paragraph 1, The above operating level is characterized by being set based on the type of processor included in the computing device or the operating speed of the artificial neural network-based detection model. method.

3. In Paragraph 2, The artificial neural network-based detection model described above includes a plurality of sub-detection models and is characterized by selecting and operating one or more of the plurality of sub-detection models according to the operation level. method.

4. In Paragraph 3, The above artificial neural network-based detection model is characterized by including one or more of a face detection model, a face feature point and expression detection model, an object classification model, and a human body pose estimation model. method.

5. In Paragraph 4, The artificial neural network-based detection model described above is characterized by predicting a risk event based on elbow angle information predicted by the human body posture estimation model. method.

6. In Paragraph 1, The above operation level is characterized by being reset during the operation of the computing device. method.

7. In Paragraph 1, A step comprising performing a security operation including logging according to the above operation level and detected risk event, method.

8. In Paragraph 7, The above log record is characterized by including one or more of the following: operation information of the computing device at the time the risk event is detected, the image data, the computing device information, user information, the type of the detected risk event, and the event detection time information. method.

9. In Paragraph 7, The above security operation is characterized by including one or more of log recording, displaying a watermark on the display screen, and displaying a security pattern on the display screen. method.

10. In Paragraph 9, The above watermark or the above security pattern is characterized by being displayed only on a part of the frame of the display screen, so that it is displayed only on the captured image among the image of the display screen and the eyes of the person viewing the display screen. method.

11. A computer program stored on a computer-readable storage medium, wherein the computer program performs operations to prevent unauthorized capturing of a display screen when executed on one or more processors, and The above operations are, An operation of measuring the hardware and software performance of the above computing device and setting an operation level; The operation of capturing a space adjacent to the front of the display screen to generate image data; and The operation of inputting the above image data and the above operation level into an artificial neural network-based detection model and predicting a risk event; is included, The above operating level is set based on the type of processor included in the computing device or the operating speed of the artificial neural network-based sensing model, and The artificial neural network-based detection model described above includes a plurality of sub-detection models and is characterized by selecting and operating one or more of the plurality of sub-detection models according to the operation level. Computer program.

12. A computing device for preventing unauthorized filming of a display screen, A processor comprising at least one core; Memory containing program code executable in the above processor; and network unit; Includes, The above processor is, Measure the hardware and software performance of the computing device to set an operation level, capture the space adjacent to the front of the display screen to generate image data, input the image data and the operation level into an artificial neural network-based detection model to predict a risk event, and perform a security operation including log recording according to the operation level and the detected risk event. The above operating level is set based on the type of processor included in the computing device or the operating speed of the artificial neural network-based sensing model, and The artificial neural network-based detection model described above includes a plurality of sub-detection models and is characterized by selecting and operating one or more of the plurality of sub-detection models according to the operation level. device.

Images