Electronic device serving as network gateway and control method thereof

Abstract

An electronic device is disclosed. The electronic device according to the present disclosure may comprise: a camera including a communication circuit for enabling communication with an external device; and an artificial intelligence processor connected to an intranet through the communication circuit and configured to execute an artificial intelligence analysis module for analyzing an image by using an artificial intelligence model, wherein the camera transmits, to the artificial intelligence processor, an image acquired through the camera, and when the artificial intelligence processor detects an event on the basis of the image, receives, from the external device, a control signal with respect to information about the event.

Description

Electronic device performing the role of a network gateway and a method for controlling the same

[0001] The present disclosure relates to an electronic device that performs the role of a network gateway and a method for controlling the same.

[0002] With the widespread adoption of video processing devices (e.g., CCTV), most buildings are equipped with them. By inputting video acquired through these devices into artificial intelligence models for analysis, abnormal objects (e.g., intruders) can be identified. Generally, since these AI models are installed on servers outside the building, video processing devices face the inconvenience of having to transmit acquired video externally. Consequently, there is a concern regarding data leakage, and there was an issue of unnecessary compliance with the Personal Information Processing Act.

[0003] Accordingly, there is a continuous demand for the development of intranet-based image processing systems capable of overcoming the limitations of the aforementioned technology.

[0004] The background description of the invention is provided to facilitate a better understanding of the present invention. The matters described in the background description should not be construed as an acknowledgment that they exist as prior art.

[0005] Meanwhile, in order to solve the aforementioned problems, the inventors of the present invention intended to develop a system capable of analyzing images captured inside a building using an intranet-based image processing device (hereinafter referred to as an artificial intelligence processor).

[0006] In particular, the inventors of the present invention recognized that by physically connecting the artificial intelligence processor to the camera, the limitations of conventional image analysis servers in image processing systems located outside the building could be overcome.

[0007] More specifically, the inventors of the present invention could expect to resolve issues related to data leakage and personal information by having an artificial intelligence processor connected to an intranet analyze the captured video and detect events such as abnormal object detection, without transmitting the captured video externally.

[0008] At this time, the inventors of the present invention recognized that problems arising in conventional image processing systems could be overcome by utilizing a camera as a network gateway.

[0009] More specifically, the inventors of the present invention have developed an electronic device capable of intranet-based image analysis that utilizes a camera as a network gateway, with respect to a new image processing device.

[0010] Furthermore, the inventors of the present invention designed a new image processing device to enable image analysis and event detection even without an external image processing server. In addition, the inventors were able to design the device so that the camera performs NAT and port forwarding functions, thereby enabling notification of an event to a central control center upon its occurrence.

[0011] Therefore, the problem that the present invention aims to solve is to provide a new image processing system through an intranet-based image processing device.

[0012] The problems of the present invention are not limited to those mentioned above, and other unmentioned problems will be clearly understood by those skilled in the art from the description below.

[0013] An electronic device according to the present disclosure includes a camera comprising a communication circuit for communicating with an external device, and an artificial intelligence processor configured to execute an image analysis module that is connected to an intranet through the communication circuit and analyzes images using an artificial intelligence model, wherein when the image analysis module detects an event based on an image acquired through the camera, the camera can transmit information about the event to the external device and receive a control signal from the external device.

[0014] Additionally, the above event may include cases where the artificial intelligence processor identifies an object detected from the image as not being a person or object that meets preset conditions.

[0015] In addition, the electronic device can transmit the video to the artificial intelligence processor in real time via RTSP (Real Time Streaming Protocol) streaming based on the IP address of the artificial intelligence processor using the camera, and detect the event based on the video transmitted in real time using the artificial intelligence processor.

[0016] In addition, the electronic device may be configured to use the camera as a network gateway to transmit the event information to an external device.

[0017] In addition, the electronic device can perform Network Address Translation (NAT) to transmit the detected event information through the camera to the external device.

[0018] In addition, when the electronic device receives a signal from the external device commanding it to provide a notification to the user in accordance with the occurrence of the event, it can perform port forwarding through the camera and transmit the signal to an internal device connected to the same network as the electronic device.

[0019] Additionally, the image analysis module and the camera may be connected via a first cable capable of TCP / IP protocol and a second cable which is a serial cable separate from the first cable, and the image may be transmitted through the first cable, and at least some control data may be transmitted through the second cable. Additionally, the image may include dynamic and still images acquired through the camera.

[0020] A control method for an electronic device including a camera and an artificial intelligence processor according to the present disclosure comprises the steps of: transmitting an image acquired through a camera via an intranet to an artificial intelligence processor configured to execute an image analysis module that analyzes the image using an artificial intelligence model; transmitting information about the event to an external device when the artificial intelligence processor detects an event based on the image; and receiving a control signal from the external device. The camera may include a communication circuit capable of communicating with the external device.

[0021] In a non-transient computer-readable recording medium that stores one or more instructions executed by a processor of an electronic device including a camera and an artificial intelligence processor of the present disclosure so as to perform an operation, the operation comprises the steps of: transmitting an image acquired through the camera via an intranet to an artificial intelligence processor configured to execute an image analysis module that analyzes the image using an artificial intelligence model; transmitting information about the event to an external device when the artificial intelligence processor detects an event based on the image; and receiving a control signal from the external device. The camera may include a communication circuit for communicating with the external device.

[0022] The present invention provides a new image processing device combining a camera and an artificial intelligence processor, which analyzes images and detects event occurrences based on an intranet, thereby reducing the risk of data leakage compared to transmitting images to an external image analysis server.

[0023] Accordingly, the present invention can improve security while complying with the Personal Information Protection Act.

[0024] Furthermore, by utilizing a camera as a network gateway, the present invention enables the operation of multiple devices, including an artificial intelligence processor existing on the same intranet, using only a single public IP.

[0025] Through this, the AI ​​processor can be accessed from the outside using only the camera's IP and port information, and network resources and installation costs can be reduced when operating multiple devices.

[0026] The effects according to the present invention are not limited to those exemplified above, and various other effects are included in this specification.

[0027] FIG. 1 is a drawing showing the operation of an electronic device according to at least one embodiment of the present disclosure.

[0028] FIGS. 2 and FIGS. 3 are schematic block diagrams of an image processing device according to at least one embodiment of the present disclosure.

[0029] FIG. 4 is a schematic block diagram of a server included in a camera according to at least one embodiment of the present disclosure.

[0030] FIGS. 5A and FIGS. 5B are drawings for illustrating the appearance of an electronic device according to at least one embodiment of the present disclosure.

[0031] FIGS. 6a and 6b are drawings for illustrating the connection of each component included in an electronic device according to at least one embodiment of the present disclosure.

[0032] FIG. 7 is a block diagram showing the configuration of an artificial intelligence processor according to at least one embodiment of the present disclosure.

[0033] FIG. 8 is a flowchart for explaining the operation of an electronic device according to at least one embodiment of the present disclosure.

[0034] FIG. 9 is a diagram illustrating a method for an artificial intelligence processor to detect an event according to at least one embodiment of the present disclosure.

[0035] FIG. 10 is a drawing for explaining that a camera according to at least one embodiment of the present disclosure performs the role of a network gateway.

[0036] FIGS. 11a and FIGS. 11b are drawings for illustrating an external device transmitting a control signal according to at least one embodiment of the present disclosure.

[0037] FIG. 12 is a drawing for explaining that a plurality of external cameras are connected to an electronic device according to at least one embodiment of the present disclosure.

[0038] FIG. 13 is a drawing for illustrating the display of a notification on a user terminal device according to at least one embodiment of the present disclosure.

[0039] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Identical or similar components regardless of drawing symbols are assigned the same reference number, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably solely for the ease of drafting the specification and do not inherently possess distinct meanings or roles. Furthermore, in describing the embodiments disclosed in this specification, if it is determined that a detailed description of related prior art could obscure the essence of the embodiments disclosed in this specification, such detailed description will be omitted. Additionally, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification; the technical concept disclosed in this specification is not limited by the attached drawings, and it should be understood that they include all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the invention.

[0040] Terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. These terms are used solely for the purpose of distinguishing one component from another.

[0041] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0042] A singular expression includes a plural expression unless the context clearly indicates otherwise.

[0043] In embodiments of the present invention, terms such as "comprising" or "having" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not excluding in advance the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0044] Additionally, in embodiments of the present invention, at least one of the plurality of components refers to all of the plurality of components, as well as each of the plurality of components excluding the remainder, or any combination thereof. Furthermore, "configured to" may be used interchangeably with, depending on the context, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." "Configured to" does not necessarily mean only that which is "specifically designed to" in hardware. Instead, in some situations, the expression "device configured to" may mean that the device is "capable of" together with other devices or components. For example, the phrase “a processor configured (or set) to perform A, B, and C” may mean a dedicated processor for performing said operations (e.g., an embedded processor), or a generic-purpose processor (e.g., a CPU or an application processor) capable of performing said operations by executing one or more software programs stored in a memory device.

[0045] One or more of the surveillance cameras, autonomous vehicles, user terminals, and servers may be linked with artificial intelligence modules, robots, augmented reality (AR) devices, virtual reality (VT) devices, devices related to 5G services, etc.

[0046] For the purpose of understanding the present invention, reference numerals have been used in the preferred embodiments illustrated in the drawings, and specific terms have been used to describe the embodiments; however, the present invention is not limited by said specific terms, and the present invention may include all components that are ordinarily conceivable by those skilled in the art.

[0047] The present invention may be represented by functional block configurations and various processing steps. These functional blocks may be implemented by various numbers of hardware and / or software configurations that execute specific functions. For example, the present invention may employ direct circuit configurations such as memory, processing, logic, look-up tables, etc., which can execute various functions by the control of one or more microprocessors or other control devices. Just as the components of the present invention may be implemented as software programming or software elements, the present invention may be implemented in programming or scripting languages ​​such as C, C++, Java, assembler, etc., including various algorithms implemented as combinations of data structures, processes, routines, or other programming configurations. Functional aspects may be implemented as algorithms executed on one or more processors. Additionally, the present invention may employ prior art for electronic configuration, signal processing, and / or data processing, etc. Terms such as "mechanism," "element," "means," and "configuration" may be used broadly and are not limited to mechanical and physical configurations. The above terms may include the meaning of a series of software processes (routines) in conjunction with processors, etc.

[0048] The specific embodiments described in this invention are examples and do not limit the scope of the invention in any way. For the sake of brevity of the specification, descriptions of prior electronic configurations, control systems, software, and other functional aspects of said systems may be omitted. Additionally, the connections of lines or connecting members between components shown in the drawings are illustrative of functional connections and / or physical or circuit connections, and may be replaced or additionally represented as various functional connections, physical connections, or circuit connections in actual devices. Furthermore, unless specifically stated as "essential," "importantly," etc., a component may not be strictly necessary for the application of the invention.

[0049] In the specification of the present invention (particularly in the claims), the use of the term "above" and similar descriptive terms may be in both singular and plural. Furthermore, where a range is described in the present invention, it is implied to include an invention applying individual values ​​belonging to said range (unless otherwise stated), and is equivalent to describing each individual value constituting said range in the detailed description of the invention. Finally, regarding the steps constituting the method according to the present invention, unless explicitly stated in a specific order or otherwise stated, said steps may be performed in a suitable order. The present invention is not necessarily limited by the order in which said steps are described.

[0050] In the present invention, the term "image" is used to encompass discontinuous video, still image, and MJPEG.

[0051] FIG. 1 is a drawing showing the operation of an electronic device according to at least one embodiment of the present disclosure.

[0052] Referring to FIG. 1, the electronic device (100) according to the present disclosure may be a device configured by combining a camera (101) and an artificial intelligence processor (102). The electronic device (100) according to the present disclosure is distinguished from a system including a conventional SaaS (Software as a Service)-based artificial intelligence processing server. A system including a conventional SaaS-based artificial intelligence processing server has a configuration in which the image from the camera is transmitted to a server equipped with an artificial intelligence model via an external network. Accordingly, personal information without consent may be leaked to the outside, and security may be vulnerable. However, the electronic device (100) according to the present disclosure, which will be described specifically below, has a configuration in which the camera (101) operates as a gateway that simultaneously functions as a router, and the artificial intelligence processor (102) is located on an internal network. Accordingly, personal information without consent is not leaked to the outside, and security can be enhanced.

[0053] Meanwhile, more specifically, the camera (101) according to the present disclosure may be a surveillance camera such as a visual camera, a thermal camera, or a special purpose camera. Additionally, the artificial intelligence processor (102) may be a device that identifies and analyzes objects included in the image. For example, the artificial intelligence processor (102) may be connected to an intranet via a communication circuit and configured to execute an image analysis module that analyzes the image using an artificial intelligence model. The artificial intelligence processor (102) may receive the image acquired through the camera (101) and analyze the image. For example, the camera (101) and the artificial intelligence processor (102) operate in a physically combined state, and as described above, image analysis by them may be performed within an intranet-based closed network environment. Accordingly, image analysis can be performed inside a building without transmitting data outside the network.

[0054] The electronic device (100) may be attached to a specific area inside the building (1). The specific area may include, but is not limited to, the central part of the ceiling and may include various locations such as the left wall and the right wall inside the building (1).

[0055] The electronic device (100) according to the present disclosure can detect abnormal objects within a building based on images acquired through a camera (101). The abnormal objects may include, for example, objects that are not people or objects under preset conditions.

[0056] When the electronic device (100) detects an abnormal object, it can transmit the fact of object detection and information about the detected object, etc., to the central control center (200) via the internet (10). The central control center (200) can perform the role of managing and controlling the electronic device (100).

[0057] A server may exist inside the central control center (200), and the server may communicate with the electronic device (100). Additionally, the server may transmit a control signal back to the electronic device (100) based on the received information.

[0058] FIGS. 2 and FIGS. 3 are schematic block diagrams of an image processing device according to at least one embodiment of the present disclosure.

[0059] In one embodiment, the image processing device may be a camera (101). The camera (101) may be an IP camera or a network camera. The camera (101) may acquire an image, compress the acquired image, and transmit it to an artificial intelligence processor (102). For example, an electronic device (100) may acquire an image through the camera (101), and use the artificial intelligence processor (102) to analyze the acquired image in real time to detect / identify / track / analyze / search for targets or moving objects, and provide appropriate information or functions to the user.

[0060] Referring to FIGS. 2 and FIGS. 3, the camera (101) may include a sensing unit (1010), a data processing unit (1030), and a communication unit (1050).

[0061] The sensor unit (1010) may be a sensing means including an optical system (110) and an image sensor (120).

[0062] The optical system (110) can optically process light from a subject. The optical system (110) may include at least one lens, such as a zoom lens that controls the field of view to narrow or widen according to the focal length, and a focus lens that focuses. In one embodiment, the lens may include at least one glass lens and / or at least one liquid lens. A liquid lens is a lens that can control zoom and focus by controlling the thickness (shape (curvature) of the liquid lens) using a fluid liquid. Using a liquid lens has the advantage of reducing the number of lenses and the size of the lens module, and also allows for fast focusing at a desired focal length.

[0063] The optical system (110) may further include an optical low-pass filter (OLPF), an infrared cut filter (IRCF), an iris for controlling light intensity, etc.

[0064] The image sensor (120) can perform the function of capturing a surveillance area to acquire an image. In one embodiment, the image sensor (120) can be implemented as a CCD (Charge-Coupled Device) sensor, a CMOS (Complementary Metal-Oxide-Semiconductor) sensor, etc. The image sensor (120) can convert an optical signal transmitted through the optical system (110) into an electrical signal.

[0065] The data processing unit (1030) may be an information processing means implemented with various number of hardware or / and software configurations that execute specific functions. For example, the data processing unit (1030) may refer to a data processing device embedded in hardware having a physically structured circuit to perform a function expressed by code or instructions included in a program. The data processing device embedded in hardware may include, for example, a microprocessor, a central processing unit (CPU), an image signal processor (ISP), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), and other processing devices, but the scope of the present invention is not limited thereto. The data processing unit (1030) may be implemented with at least two of the above-described processing devices and processors as a single integrated configuration (e.g., a single chip), or each as an independent configuration (e.g., multiple chips).

[0066] In one embodiment, the data processing unit (1030) may further include a neural processing unit (NPU). In one embodiment, the neural processing unit may be included in the data processing unit (1030) in a form inherent within the aforementioned processing unit and processor. The neural processing unit is a processor specialized in processing artificial intelligence models, and the artificial intelligence model may be generated through machine learning. The artificial intelligence model may include a plurality of artificial neural network layers. The artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may additionally or substantially include a software structure.

[0067] In one embodiment, the artificial intelligence model can be trained by collecting and labeling suitable training data for determining object detection / classification / recognition, etc., and applying it to an artificial neural network. The trained artificial intelligence model can be mounted on a camera (101) and used for inference operations such as object detection / classification / recognition through a neural network processing device of the camera (101) or the aforementioned data processing device.

[0068] In one embodiment, the artificial intelligence model can be further trained using a cloud server to suit the monitoring environment or user requirements even after being installed once on the camera (101), and the newly trained model can be redeployed to the camera (101) for use.

[0069] A neural network processing unit may be mounted on the camera (101) in the form of at least one hardware chip. For example, the neural network processing unit may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or may be manufactured as part of a general-purpose processor (CPU) or a graphics-dedicated processor (GPU) and mounted on the camera (101). Additionally, the neural network processing unit may be implemented as a software module. If the neural network processing unit is implemented as a software module (or a program module containing instructions), the software module may be stored on a non-transitory computer-readable media. In this case, at least one software module may be provided by an operating system (OS) or by an application. The data processing unit (1030) may include an image processing unit (130), a streaming unit (140), an analysis unit (150), an event unit (160), a system management unit (170), and a camera control unit (180).

[0070] The image processing unit (130) can improve image quality by adjusting the brightness, contrast, color amount, contrast ratio, sharpness, etc. of the image. Each parameter value for image quality improvement can be fixed as an initial value or adjusted by the user and pre-set. The image processing unit (130) can perform image signal processing for image quality improvement, such as noise reduction, gamma correction, color filter array interpolation, color matrix, color correction, and color enhancement. The image processing unit (130) can apply various methods for image quality improvement, such as edge enhancement algorithms, histogram-based image quality improvement algorithms such as histogram equalization or histogram stretching, and hue control algorithms. In one embodiment, the image processing unit (130) may be an image signal processor (ISP). The image processing unit (130) may include an exposure control unit (1211), a backlight correction unit (1312), a balance correction unit (1313), a gamma correction unit (1314), a sharpness control unit (1315), a shake correction unit (1316), a focus control unit (1317), a zoom control unit (1318), a filter control unit (1319), an OSD control unit (1320), and a noise control unit (1321).

[0071] The exposure control unit (1211) can control the brightness of the image by controlling the amount of light entering the image sensor (120) using at least one of a shutter, an iris, and gain or ISO sensitivity. The exposure control unit (1211) can control the amount of light by adjusting the shutter speed. The exposure control unit (1211) can control the amount of light by adjusting the degree of opening and closing of the iris. The exposure control unit (1211) can control the degree of light amplification by adjusting the gain or ISO sensitivity. In one embodiment, the exposure control unit (1211) can control the shutter, the iris, and the gain / ISO sensitivity according to user settings. In one embodiment, when the function is set, the exposure control unit (1211) can perform an auto exposure algorithm according to the illuminance of the environment in which the camera is installed.

[0072] The backlight correction unit (1312) can improve the quality of an image by applying a backlight correction algorithm to the image so that dark areas are clearly visible when the image is too bright overall. The backlight correction algorithm may include a Back Light Compensation (BLC) algorithm, a Wide Dynamic Range (WDR) algorithm, a Digital Wide Dynamic Range (D-WDR) algorithm, and a High Dynamic Range (HDR) algorithm. The backlight correction unit (1312) can increase the dynamic range using the WDR algorithm, the D-WDR algorithm, and the HDR algorithm, thereby enabling both bright and dark areas in the image to be clearly visible simultaneously. In one embodiment, the backlight correction unit (1312) can block backlighting in a specific area to make the image of other areas appear brighter. For example, when car headlights shine strongly at a dark parking lot entrance or a gas station entrance at night, only the headlight light can be blocked to identify the vehicle license plate. The function of the backlight correction unit (1312) can be set by the user (on / off).

[0073] The balance correction unit (1313) can improve the color identification ability of an image by adjusting the white balance of the image to correct color differences according to the light source (illumination). In one embodiment, the balance correction unit (1313) can adjust the white balance appropriately for the selected light source according to user settings. In one embodiment, the balance control unit (1313) can perform an Auto White Balance algorithm according to the environment around the camera when its function is set. In one embodiment, the balance control unit (1313) can adjust the white balance of the image to a white balance value manually set by the user.

[0074] The gamma correction unit (1314) can increase visibility by changing the contrast of the image through gamma correction.

[0075] The sharpness control unit (1315) can increase the resolution by adjusting the sharpness of the subject boundary (contour).

[0076] The shake correction unit (1316) can stabilize the image by applying a shake correction algorithm to the image to correct image shake caused by camera shake. The shake correction algorithm may include an Optical Image Stabilization (OIS) algorithm, an Electronic Image Stabilization (EIS) algorithm, and a Digital Image Stabilization (DIS) algorithm.

[0077] The focus control unit (1317) can control the focus of the camera using a focus control algorithm. In one embodiment, the focus control unit (1317) can move the position of the focus lens by controlling the focus motor according to a control signal for focus control. The focus control algorithm may include known autofocus algorithms such as a hill climbing method. For example, the focus control unit (1317) can move the position of the focus lens from a near distance to a far distance (infinity) in real time, compare the focus values ​​before and after to check the increase or decrease in the focus value, the tilt, and the change in the tilt, determine the peak point with the largest focus value as the in-focus position, and control the driving of the focus motor to move the focus lens to the in-focus position. In one embodiment, the focus control unit (1317) can control the focus by controlling the curvature of the liquid lens according to a control signal for focus control.

[0078] The zoom control unit (1318) can perform optical zoom and / or digital zoom functions according to a control signal for controlling the zoom magnification. In one embodiment, the zoom control unit (1318) can adjust the zoom magnification by moving the position of the zoom lens to zoom in or zoom out. In one embodiment, the zoom control unit (1318) can control the zoom magnification by controlling the curvature of the liquid lens. In one embodiment, the zoom control unit (1318) can provide a more magnified image through digital zoom exceeding the optical zoom. The angle of view or field of view of the camera can be controlled by the zoom magnification.

[0079] The filter control unit (1319) can improve image identification capabilities during the day and night by adjusting the infrared cut filter (IRCF). The filter control unit (1319) can block infrared rays and allow visible light to pass through by positioning the infrared cut filter (IRCF) in front of the image sensor (120). The filter control unit (1319) can allow infrared rays to pass through in addition to visible light by removing the infrared cut filter (IRCF). The filter control unit (1319) can perform a day / night mode function by automatically turning the infrared cut filter (IRCF) on / off according to the ambient light level.

[0080] The OSD (On-screen display) control unit (1320) can overlay a mask of text and / or polygons on the image. For example, the OSD (On-screen display) control unit (1320) can display the date, time, manufacturer, etc. at a predetermined location on the image.

[0081] The noise control unit (1321) can perform a digital noise reduction algorithm to reduce color noise in an image captured in a low-light environment. The noise reduction algorithm may include 2D-NR (Noise Reduction) and 3D-NR.

[0082] The streaming unit (140) can compress the video received from the video processing unit (130) and provide it to an external device via a network. The streaming unit (140) may include an encoder (1411), an audio processing unit (1412), and an RTSP server (1413).

[0083] The encoder (1411) can encode the image and compress the image. The encoder (1411) can compress the raw image input from the image sensor (120) or the image processed by the image processing unit (130). In one embodiment, the raw image may be used for artificial intelligence (AI) algorithms, etc.

[0084] The encoder (1411) can compress video using an intra-frame compression method and / or an inter-frame compression method. Depending on the network environment or monitoring purpose, the encoder (1411) can compress video into compression formats such as H.264, H.265 (HEVC (High Efficiency Video Coding)), JPEG, and MPEG-4.

[0085] The encoder (1411) can compress video according to encoding settings to generate at least one video stream. Encoding settings may include compression format, video quality (high quality / medium quality / low quality), resolution, bit rate, frame rate, etc. The encoder (1411) may compress video using a constant bit rate (CBR) or a variable bit rate (VBR). A profile of the video stream may be determined according to the encoding settings. The profile may represent the specifications of the video stream transmitted from the camera (101) to an external device. The profile may include setting values ​​such as compression format, video quality (high quality / medium quality / low quality), resolution, frame rate, bit rate, etc. For example, the profile may include setting values ​​such as "MJPEG, low quality, 640X480, 5fps, 2Mbps". In one embodiment, a profile of a video stream output from an encoder (1411) may be pre-set according to the network environment and the video receiving device (external device). For example, a profile suitable for streaming purposes may be pre-set, such as a profile for a video stream for storage, a profile for a video stream for monitoring screen output, a profile for a video stream for a mobile device, or a profile for a video stream for a web viewer. The encoder (1411) can compress the video using the setting values ​​of the selected profile.

[0086] The audio processing unit (1412) includes an audio codec and can generate at least one audio stream by processing and compressing audio input from an audio sensor and / or a stored sound source. Depending on the microphone structure, the audio sensor may be a dynamic microphone, a condenser microphone, a ribbon microphone, etc., and depending on the directionality, a directional microphone, an omnidirectional microphone, a super-directional microphone, etc., may be used. The audio sensor may be provided separately from the image sensor (120) or integrally, such as being embedded in the image sensor (120).

[0087] The RTSP server (1413) may be a server operated for real-time video streaming. The RTSP server (1413) may transmit video streams and audio streams to external devices connected to the RTSP port via a network according to the Real-time Streaming Protocol (RTSP).

[0088] The analysis unit (150) can analyze the image processed by the image processing unit (130) in real time. The analysis unit (150) can perform background area detection, foreground and object detection, object counting, camera tampering detection, face detection, etc. Additionally, the analysis unit (150) can calculate brightness, color, texture, and shape information of the image. The analysis unit (150) can analyze audio input from an audio sensor in real time. The results of the image and / or audio analysis can be generated as metadata. In one embodiment, the analysis unit (150) can detect / recognize / identify objects using artificial intelligence (AI) based technology.

[0089] The analysis unit (150) may include a motion detection unit (1511), an audio detection unit (1512), an audio classification unit (1513), a tempering detection unit (1514), and a defocus detection unit (1515).

[0090] The motion detection unit (1511) can detect the motion of an object within an image. The motion detection unit (1511) can detect the motion of an object from an image using a motion detection (MD) algorithm. In one embodiment, the motion detection unit (1511) can detect motion using an artificial neural network (ANN).

[0091] The audio detection unit (1512) can detect audio input from the audio sensor.

[0092] The audio classification unit (1513) can classify the detected audio. In one embodiment, the audio can be classified into vocal, which is a sound produced from a person's throat, and non-vocal, which is other sounds. For example, vocal can be classified into conversation, scream, crying, etc., and non-vocal can be classified into footsteps, glass breaking, explosion, crash, gunshot, etc. It goes without saying that vocal and non-vocal are not limited to the types mentioned above and can be set in various ways depending on the monitoring area and system design. In one embodiment, the audio classification unit (1513) can learn audio features and classify the audio using an Artificial Neural Network (ANN).

[0093] The tempering detection unit (1514) can detect camera movement or camera obstruction situations that last for a predetermined period of time. For example, the tempering detection unit (1514) can detect situations where spray is applied to the lens, where the lens is covered, etc.

[0094] The defocus detection unit (1515) can detect focus distortion.

[0095] The event unit (160) may receive video and / or audio from the analysis unit (150). The event unit (160) may receive metadata indicating the analysis results of the video and / or audio from the analysis unit (150). The event unit (160) may detect an event based on the metadata and notify the detection of the event. The event unit (160) may generate an event rule to perform an action on the device when a specific event occurs. The event rule may include conditions and actions. The conditions may be detailed rules for detecting the event. The actions may be operations performed when the occurrence of the event is determined. The event unit (160) may associate an action to be taken in response to the occurrence of the event. In one embodiment, the event conditions and the action associated with the event may be predefined by the user.

[0096] The event unit (160) may include an event generation unit (1611), a handover unit (1612), an alarm unit (1613), an FTP upload unit (1614), a storage control unit (1615), and an MQTT unit (1616).

[0097] The event generation unit (1611) can generate conditions by combining various situations detected as a result of analyzing video and / or audio. The event generation unit (1611) can detect events based on conditions set according to the type of event.

[0098] In one embodiment, the event generating unit (1611) may generate a video event when the analysis result of the video satisfies a set event condition. Here, the video event may include the appearance and / or disappearance of an object within the screen, the occurrence of a specific image (e.g., the appearance of a face that cannot be recognized) by the user, the change in screen color, the detection of motion of an object, the detection of motion by the user within a specific area, the detection of motion by the user in a specific direction, the turning off of the video, tampering, the distortion of focus, etc.

[0099] In one embodiment, the event generating unit (1611) may generate an audio event when the result of analyzing the audio satisfies a set event occurrence condition. The audio event may include events set by the user, such as the occurrence of a specific audio signal such as conversation, scream, crying, or shouting; the occurrence of abnormal audio signals such as footsteps, alarm, crash, car tire skid, glass breaking sound, explosion, or gunshot; or the occurrence of a voice exceeding a threshold value.

[0100] In one embodiment, the event generation unit (1611) may generate an event when a combination of video analysis results and / or audio analysis results satisfies a set event occurrence condition. It goes without saying that video events and audio events are not limited to the types described above and can be set in various ways depending on the monitoring area and system design.

[0101] The handover unit (1612) can transmit the generated event or event generation conditions to another camera, or receive the event or event generation conditions from another camera, so that the camera (101) can perform a set action when an event occurs, such as panning, tilting, zooming, or automatic object tracking.

[0102] The alarm unit (1613) can output an alarm signal when an event occurs. The alarm unit (1613) can trigger input from various sensors outside the camera or output to an actuator to output an alarm signal. The alarm unit (1613) can notify of the occurrence of an event through audio playback, email transmission, etc.

[0103] The FTP upload unit (1614) can upload the video and / or audio in which the event is detected to a File Transfer Protocol (FTP) server on the network when an event occurs. The video may include still images and video images. In one embodiment, the FTP upload unit (1614) can upload the video and / or audio in which the event is detected to an FTP server at a set upload period and a set upload speed when an event occurs.

[0104] The storage control unit (1615) can store (record) video and / or audio in a storage device. In one embodiment, the storage control unit (1615) can store video and / or audio and metadata received from the analysis unit (150) in a storage device. The metadata may include object information (movement, sound, intrusion into a designated area, etc.), object identification information (person, car, license plate, face, hat, clothing, etc.) detected from the video and / or audio, and detected location information (coordinates, size, etc.). In one embodiment, when an event occurs, the storage control unit (1615) can store video and / or audio and metadata for a predetermined period of time before and after the event in a storage device. The storage control unit (1615) can store video according to a set profile.

[0105] The storage device can store various programs and data required for the operation of the camera (101). Data reading, recording, modification, deletion, and updating can be performed by each component including the image processing unit (130) of the storage device. Additionally, the storage device can store a neural network model (e.g., a deep learning model) generated through a learning algorithm for data recognition / classification. The storage device may include internal memory and / or external storage media such as an SD card. The storage device may include a Video Management System (VMS), a Network Video Recorder (NVR), etc.

[0106] The MQTT unit (1616) can transmit (notify) events or receive notifications of event occurrences using the MQTT (Message Queuing Telemetry Transport) protocol. In one embodiment, the MQTT unit (1616) includes an MQTT client and can transmit events in real time by publishing MQTT messages to an MQTT broker (server) using the MQTT protocol through the MQTT client. In one embodiment, the MQTT unit (1616) can subscribe to MQTT messages published to the MQTT broker (server) to receive notifications of event occurrences in real time and start video recording.

[0107] The system management unit (170) may include a firmware upgrade unit (1711), a system information management unit (1712), a power management unit (1713), an RTC management unit (1714), and a network management unit (1715).

[0108] The firmware upgrade unit (1711) can upgrade the firmware (F / W) of the camera. The firmware upgrade unit (1711) can upgrade the old version firmware by receiving new firmware (F / W) from the firmware server via wired or wireless connection.

[0109] The system information management department (1712) can manage product information such as the camera model name and serial number.

[0110] The power management unit (1713) can convert AC or DC power supplied from an external power source into power required for the operation of each component of the camera. The power management unit (1713) may include an auxiliary power source, such as a rechargeable built-in or replaceable battery. The power management unit (1713) can manage power according to the class of the Ethernet Power over Ethernet (PoE). The power management unit (1713) can manage and control the auxiliary power source and the external power source in maximum power mode and low power mode.

[0111] The RTC management unit (1714) can manage the camera's internal time by synchronizing with a network server such as NTP.

[0112] The network management unit (1715) can manage general network functions such as IP address settings, port settings, and DDNS server settings.

[0113] The camera control unit (180) can control the camera's posture by moving the camera to a physical position corresponding to the Pan / Tilt / Rotate / ZoomFocus value according to the object's motion information, user input, or preset. The camera control unit (180) may include a pan control unit (1811), a tilt control unit (1812), a rotation control unit (1813), a preset control unit (1814), and a sequence action control unit (1815).

[0114] The fan control unit (1811) can control the rotation of the camera in the horizontal direction. In one embodiment, the fan control unit (1811) can rotate the camera in the horizontal direction by controlling the driving of the fan motor based on the fan value.

[0115] The tilt control unit (1812) can control the rotation of the camera in the vertical direction. In one embodiment, the tilt control unit (1812) can rotate the camera in the vertical direction by controlling the driving of the tilt motor based on the tilt value.

[0116] The rotation control unit (1813) can control the rotation of the camera in a clockwise or counterclockwise direction of the image. The rotation control unit (1813) can rotate the lens in a clockwise or counterclockwise direction of the image with the center of the camera as the axis.

[0117] The preset control unit (1814) can quickly move the camera to a physical position corresponding to a pre-specified Pan / Tilt / Rotate / ZoomFocus value (preset). The preset control unit (1814) can change the camera's orientation by moving or rotating the camera according to at least one specified preset.

[0118] The sequence action control unit (1815) can control the camera's operation repeatedly by scheduling operations that are pre-specified, such as presets.

[0119] Referring again to FIG. 2, the communication unit (1050) may be an interface that transmits a stream by connecting to a server (2000) and an external device for wired or wireless communication. The communication unit (1050) may be configured to include TCP / IP, HTTP(S) / RTP / RTSP, FTP, MQTT protocols, etc.

[0120] The electronic device (100) can use the camera (101) as a network gateway by using the communication unit (1050). For example, the electronic device (100) can transmit data to an internal device connected to the intranet via the communication unit (1050) of the camera (101), or transmit it to a central control center (200) via the internet. Alternatively, the electronic device (100) can receive data from the central control center (200) via the internet and transmit control signals to an internal device connected to the intranet based on the data.

[0121] In order for the electronic device (100) to transmit and receive data to and from internal devices connected to the intranet and a central control center (200) in this manner, the NAT function and port forwarding function of the communication unit (1050) can be utilized. A detailed explanation regarding NAT and port forwarding will be described later based on FIG. 10.

[0122] To prevent the features of the present embodiment from being obscured, only the components related to the present embodiment are illustrated. Accordingly, it can be understood by those skilled in the art related to the present embodiment that other general components may be included in addition to the components illustrated in FIGS. 2 and FIGS. 3.

[0123] FIG. 4 is a schematic block diagram of a server embedded in or connectable to a camera according to at least one embodiment of the present disclosure.

[0124] Referring to FIG. 4, a server (2000) according to one embodiment may include an FTP server (2011), a web server (2013), an authentication server (2015), etc. The server (2000) may be provided independently on a network separate from the camera or embedded in the camera, but for convenience in this disclosure, it is described as being embedded in the camera (101).

[0125] The FTP server (2011) can receive video and / or audio in which an event is detected from the FTP upload unit (1614) of the video processing device when an event occurs. Additionally, the FTP server (2011) can store firmware data uploaded by a user for firmware updates.

[0126] The web server (2013) can provide HTTP(S) access to the image processing device and external device, or be embedded in the image processing device or server to provide HTTP(S) access to external device.

[0127] The authentication server (2015) can perform user authentication when attempting to connect to an external device's image processing device using RTP / RTSP / HTTP / TCP, etc.

[0128] FIG. 5a is a drawing for explaining the appearance of an electronic device according to at least one embodiment of the present disclosure.

[0129] Referring to FIG. 5a, the electronic device (100) may include a camera (101) and an artificial intelligence processor (102). The camera (101) and the artificial intelligence processor (102) may be physically connected by a connecting cable (12). A specific example of the connecting cable (12) will be described later based on FIG. 6a.

[0130] FIG. 6a is a drawing for illustrating the connection of each component included in an electronic device according to at least one embodiment of the present disclosure.

[0131] Referring to FIG. 6a, a sensor unit (1010) and a data processing unit (1030) may be connected to the inside of the camera (101) of the electronic device (100) via a first cable (13-1), and a data processing unit (1030) and an artificial intelligence processor (102) may be connected via the first cable (13-1) and a second cable (13-2).

[0132] As described above, the sensor unit (1010) may be a sensing means including an optical system (110) and an image sensor (120). The sensor unit (1010) may be described using terms such as "camera lens module." Since a detailed description of the sensor unit (1010) has been described above through FIGS. 2 and FIGS. 3, a redundant description is omitted.

[0133] As described above, the data processing unit (1030) is an information processing means implemented with various number of hardware or / and software configurations that execute specific functions, and may include a central processing unit (e.g., CPU). Since a detailed description of the data processing unit (1030) has been described above through FIGS. 2 and FIGS. 3, a redundant description is omitted.

[0134] The first cable (13-1) may include a cable capable of TCP / IP protocol. Specifically, the first cable (13-1) may be a cable modified from a UTP cable to reduce its size so that it can be contained inside the camera (101). Thus, the first cable (13-1) may be used to connect the sensor unit (1010) and the data processing unit (1030) inside the camera (101). Additionally, the first cable (13-1) may be used to connect the camera (101) and the artificial intelligence processor (102).

[0135] The second cable (13-2) may include a serial cable. The second cable (13-2) may include a serial cable and can transmit and receive control signals between devices by sequentially transmitting data one bit at a time. At least some of the control data may be transmitted through the second cable.

[0136] The electronic device (100) can acquire an image from the sensor unit (1010) and transmit the acquired image to the data processing unit (1030) via the first cable (13-1). The electronic device (100) can transmit the image transmitted to the data processing unit (1030) to the artificial intelligence processor (102) via the first cable (13-1). The artificial intelligence processor (102) can analyze the received image and transmit the image analysis result to the data processing unit (1030) via the first cable (13-1).

[0137] Based on FIG. 6a, the aforementioned connecting cable (12) may include a first cable (13-1) and a second cable (13-2). However, the connecting cable (12) is not limited thereto and may include an Ethernet cable or a USB cable, etc.

[0138] Meanwhile, the electronic device (100) of the present disclosure is not limited to the form in which the camera (101) and the artificial intelligence processor (102) described based on FIGS. 5a and 6a are each installed on a wall surface in a form connected by a cable. Other forms of the appearance of the electronic device (100) will be described based on FIG. 5b.

[0139] Referring to FIG. 5b, the present disclosure may exist in the form of an integrated electronic device (100a) in which a camera (101a) and an artificial intelligence processor (102a) are located inside a single housing.

[0140] The internal configuration of the integrated electronic device (100a) may include the same configuration as described in FIG. 6a. Specifically, the sensor unit (1010), data processing unit (1030), and artificial intelligence processor (102) of FIG. 6a may correspond to the sensor unit (1010a), data processing unit (1030a), and artificial intelligence processor (102a) of FIG. 6b, respectively. Additionally, the first cable (13-1) and the second cable (13-2) of FIG. 6a may correspond to the first cable (13-1a) and the second cable (13-2a) of FIG. 6b, respectively.

[0141] In one embodiment, the integrated electronic device (100a) includes a plurality of camera lenses so as to acquire an image with a minimized blind spot. For example, the plurality of camera lenses may be arranged in an omnidirectional manner to minimize the blind spot.

[0142] In one embodiment, the components included in the integrated electronic device (100a) may exist in a stacked form. For example, a sensor unit (1010a), a data processing unit (1030a), and an artificial intelligence processor (102a) may be connected by multiple wires and exist in a stacked structure inside the housing of the integrated electronic device (100a). Such a stacked structure can reduce the device volume by minimizing unnecessary empty space in the integrated electronic device (100a).

[0143] In addition, the aforementioned artificial intelligence processor (102, 102a) may exist in a detachable structure. In one embodiment, the artificial intelligence processor (102, 102a) can be mounted and detached without tools through a mounting bracket inside the housing and a one-touch connector, and the electronic device (100) or the integrated electronic device (100a) can recognize the detached state of the artificial intelligence processor (102, 102a) and control the switching of the artificial intelligence analysis mode. According to this configuration, rapid recovery may be possible simply by replacing the module even in the event of a failure of the artificial intelligence processor (102, 102a). In addition, an upgrade can be easily performed by easily replacing it with a higher specification module according to processing performance requirements.

[0144] FIG. 7 is a block diagram showing the configuration of an artificial intelligence processor according to at least one embodiment of the present disclosure.

[0145] Referring to FIG. 7, the artificial intelligence processor (102) may include memory (1021), GPU (1022), NPU (1023), video engine (1024) and CPU (1025).

[0146] In order to prevent the features of the present embodiment from being obscured, only the components related to the present embodiment are shown in the configuration illustrated in FIG. 7. Therefore, it can be understood by those skilled in the art related to the present embodiment that other general components may be included in addition to the components illustrated in FIG. 7.

[0147] The memory (1021) may be implemented as memory embedded in the artificial intelligence processor (102) for data storage purposes (e.g., volatile memory (e.g., semi-permanent memory such as RAM (random access memory)), non-volatile memory (e.g., permanent memory such as ROM (read-only memory)), flash memory, hard drive or solid-state drive, etc.), or as memory that can be attached to the artificial intelligence processor (102) (e.g., memory card, external memory, etc.).

[0148] Instructions may be stored in the memory (1021). The artificial intelligence processor (102) may perform the operation of the artificial intelligence processor (102) according to various embodiments of the present disclosure by executing the instructions in the memory (1021) individually or collectively. Additionally, programs and data for operating the artificial intelligence processor (102) may be stored in the memory (1021). For example, the memory (1021) may store one or more software applications, such as operating system (or system) software applications, firmware software applications, driver software applications, plugin (e.g., add-in, add-on, and / or applet) software applications, and / or any other suitable software applications.

[0149] Meanwhile, in the present disclosure, the term memory (1021) may be used to include memory (1021), ROM, RAM within the artificial intelligence processor (102), or a memory card (e.g., micro SD card, memory stick) mounted in the artificial intelligence processor (102). In addition, various information necessary within the scope of achieving the purpose of the present disclosure may be stored in the memory (1021), and the information stored in the memory (1021) may be updated as it is received from an external device or input by a user.

[0150] The memory (1021) may have substantially the same configuration as the storage device described above based on FIGS. 2 and FIGS. 3.

[0151] The GPU (1022) is a graphics-dedicated processor and may be a processor designed to include multiple parallel processing units to analyze large volumes of video data in real time. The GPU receives high-resolution frames (e.g., 4K or higher) as a single or multiple stream and can quickly process operations that require high parallelism at the pixel level, such as object detection and tracking, feature vector extraction, spatial and frequency filtering, and image quality enhancement.

[0152] For example, since the GPU (1022) inputs and outputs data to and from system memory or high-bandwidth dedicated graphics memory (HBM / GDDR, etc.) via Direct Memory Access (DMA), it can read video frames directly without passing through the CPU and perform large-scale matrix operations (MAC, Tensor operations, etc.). Accordingly, the latency per frame is reduced, and CPU resources can be concentrated on control and management tasks such as event logic, network transmission and reception, and user interface. In addition, the GPU can utilize matrix operation specialized computation units (e.g., Tensor / MMA cores) provided by the latest deep learning frameworks to perform inference of trained artificial intelligence models, such as Convolutional Neural Networks (CNN) and Vision Transformers (ViT), in real time, generate metadata from the results, and transmit them to a storage unit or an external server. In the examples described below, the external server is described using the same term as the external device.

[0153] The NPU (1023) may be a neural network processing unit. The neural network processing unit may be a device included in the data processing unit (1030) in a form inherent within the aforementioned processing unit and processor. Since the specific description of the NPU (1023) is the same as that described above in FIGS. 2 and FIGS. 3, a redundant description is omitted. The electronic device (100) can analyze images and identify abnormal objects using the GPU (1022) and the NPU (1023).

[0154] In addition, although the GPU (1022) was depicted as a separate component in the example described above, the form of the GPU (1022) is not limited to this, and the GPU (1022) may operate in conjunction with the CPU (1025).

[0155] The video engine (1024) may include a dedicated multimedia acceleration block separately provided within a system-on-chip (SoC) or a graphics card. The video engine (1024) can process high-resolution video streams in real time by performing encoding, decoding, and necessary post-processing of video data with dedicated hardware. The video engine (1024) may perform real-time compression and decompression of video streams, for example, by including a hardware encoder and decoder.

[0156] Specifically, the video engine (1024) can decode the encoded video (EIMG) received from the camera (101) by a decoder and analyze the decoded video (DIMG) through an artificial intelligence model.

[0157] The CPU (1025) is a central processing unit and can control the overall operations of the artificial intelligence processor (102). For example, the CPU (1025) can cause other components of the artificial intelligence processor (102) to perform various operations by executing instructions stored in memory (1021). For example, the CPU (1025) can control the operation of the artificial intelligence processor (102) by being operatively connected to memory (1021). Additionally, the CPU (1025) can control the operation of the artificial intelligence processor (102) according to the present disclosure by executing one or more instructions stored in memory (1021).

[0158] The CPU (1025) may be implemented as one or more IC (integrated circuit (or circuitry)) chips and may perform various data processing operations. The CPU (1025) may include at least one electrical circuit and may process instructions (or programs, data, etc.) stored in memory (1021) individually or collectively.

[0159] In the example described above, each configuration is depicted as adopting a Direct Memory Access (DMA) method that does not go through the CPU (1025), but the communication method of each configuration is not limited to this.

[0160] Hereinafter, with reference to FIG. 8, the process of a building security system operating based on communication between a camera (101) and an artificial intelligence processor (102) is explained.

[0161] FIG. 8 is a flowchart for explaining the operation of an electronic device according to at least one embodiment of the present disclosure.

[0162] Referring to FIG. 8, in operation S810, the electronic device (100) can transmit an image acquired through the camera (101) to an artificial intelligence processor (102).

[0163] The camera (101) is located inside the building and can acquire images of the building's interior in real time. The acquired images may include shapes of people, animals, objects, etc. inside the building. Additionally, multiple cameras (101) may exist inside the building.

[0164] The camera (101) includes a communication circuit for communicating with an external device and can transmit the acquired image to an artificial intelligence processor (102) in real time.

[0165] In one embodiment, the camera (101) can transmit video to the artificial intelligence processor (102) in real time via RTSP (Real Time Streaming Protocol) streaming based on the IP address of the artificial intelligence processor (102). RTSP (Real Time Streaming Protocol) streaming may be a control protocol used when streaming real-time media data, such as video or audio, over a network. That is, the artificial intelligence processor (102) can receive and analyze video in real time without downloading video from the camera (101).

[0166] In operation S820-Y, S830, when the electronic device (100) detects an event based on an image using an artificial intelligence processor (102), it can transmit information about the event to an external device.

[0167] The artificial intelligence processor (102) can detect events based on real-time video transmitted from the camera (101).

[0168] An event may include cases where the artificial intelligence processor (102) identifies an abnormal object. For example, an event may include cases where the artificial intelligence processor (102) identifies an object detected from a received image as not being a person or object that meets preset conditions. In one embodiment, the artificial intelligence processor (102) may generate metadata (MD) based on the received image data and generate an event based on the metadata (MD). For example, the information processing means may analyze the image to recognize an object, extract metadata (MD) for the object, and combine and store the event image containing the object with the metadata (MD). The information processing means may perform image analysis, object recognition and extraction, object classification and attribute determination, and metadata extraction using a deep learning image analysis algorithm, etc. The metadata (MD) may include image-based metadata such as blob images of motion regions and background models.

[0169] Metadata (MD) may include information extracted from detection data. For example, if the detection data is image data, the metadata (MD) may include information such as image identification information, the appearance / disappearance of objects (people, cars, etc.), object attributes (e.g., object information such as type, color, size, location, shape, movement, trajectory), object situation information (e.g., getting on or off), the occurrence of specific images (e.g., appearance of specific targets), and face detection. If the detection data is audio data, the metadata (MD) may include information such as the occurrence of abnormal sound sources (e.g., car tire friction sound (skid), glass breaking sound, alarm sound, collision sound, shouting, screaming, crying sound, etc.) and the occurrence of sounds exceeding a threshold.

[0170] Referring to FIG. 9, the artificial intelligence processor (102) can detect events from an image using an artificial intelligence model (902).

[0171] As described above, the artificial intelligence processor (102) can identify abnormal objects by analyzing the image through image codec technology and an artificial intelligence model.

[0172] In one embodiment, the artificial intelligence processor (102) can input the received image (901) into the artificial intelligence model (902). The artificial intelligence processor (102) can analyze the image through the artificial intelligence model (902) to identify whether an event has occurred, that is, whether an abnormal object such as an intruder exists (903).

[0173] The artificial intelligence processor (102) can transmit information about the event to the central control center (200) when an event occurs.

[0174] In one embodiment, information can be transmitted to a central control center (200) via a camera (101). The electronic device (100) according to the present disclosure may be configured to transmit event information to an external device using the camera (101) as a network gateway. That is, the camera (101) can be utilized as a network gateway by performing a NAT function.

[0175] FIG. 10 is a drawing for explaining that a camera according to at least one embodiment of the present disclosure performs the role of a network gateway.

[0176] Referring to FIG. 10, the electronic device (100) can perform Network Address Translation (NAT) to transmit event information detected through the camera (101) to an external device.

[0177] Specifically, a device connected to a network can communicate using a private IP (Private IP, 1001) or a public IP (Public IP, 1002).

[0178] A private IP (1001) is an IP address used only within an internal network (e.g., a local network such as a home or company) and cannot be directly identified or accessed from an external network such as the Internet. Each device connected to an internal network can be assigned a unique private IP address within that network.

[0179] On the other hand, a public IP (1002) is an IP address uniquely assigned worldwide, which can be directly identified on the internet and communicate with an external network.

[0180] Network Address Translation (NAT) may be an operation that converts a private IP address into a public IP address so that a device using a private IP (1001) can communicate with an external network. Therefore, NAT enables multiple internal devices on an intranet to communicate with the outside world through a single public IP (1002), and security can be improved as internal information is not exposed to the outside.

[0181] In operation S840, the electronic device (100) can receive a control signal from an external device.

[0182] The central control center (200) can transmit a control signal to the electronic device (100) based on receiving event information from the electronic device (100).

[0183] In one embodiment, the central control center (200) can analyze event information and determine whether to send a notification to the user. For example, the central control center (200) can analyze event information and, if it identifies the detected abnormal object as an object such as a person with mobility impairment or a robber, decide to send a notification to the user.

[0184] When the central control center (200) decides to send a notification to the user, it may send a control signal to the electronic device (100). The control signal may include a command in which the central control center (200) instructs the electronic device (100) to display an event occurrence notification on the display of an internal device (e.g., a user terminal device) present within the building.

[0185] Accordingly, when the electronic device (100) receives a signal from an external device commanding it to provide a notification to the user upon the occurrence of an event, it can perform port forwarding through the camera (101) to transmit the signal to an internal device connected to the same network as the electronic device (100).

[0186] Referring to FIG. 10, the camera (101) can be utilized as a network gateway by performing a port forwarding function.

[0187] Port forwarding may be a network configuration method that enables an external device to access an internal device by forwarding a request received from a central control center (200) to a designated device in the internal network. That is, through port forwarding, the electronic device (100) can forward a request received from an external device using the public IP (1002) to a designated device in the internal network using the private IP (1001) by mapping a specific port of the public IP (1002) to the private IP and port of the internal device.

[0188] FIGS. 11a and FIGS. 11b are drawings illustrating an external device transmitting a control signal according to at least one embodiment of the present disclosure. The following description is based on FIG. 11a, but the present disclosure is not limited thereto, and the integrated electronic device (100b) shown in FIG. 11b can also receive a control signal from an external device in the same manner as FIG. 11a.

[0189] Referring to FIG. 11a, as described above, the electronic device (100) can transmit a signal requesting an internal device connected to the same network as the electronic device (100) to display a notification through port forwarding based on a received control signal.

[0190] In one embodiment, the internal device may be a laptop PC (1100). The electronic device (100) may transmit a signal requesting a notification display to the laptop PC (1100) in accordance with a control signal, thereby causing a notification to be displayed on the corresponding display. Accordingly, a warning notification such as "abnormal object detection" may be displayed on the display of the laptop PC (1100), and a real-time video may be displayed along with the warning notification. Based on this, the user may take action regarding the detection of an abnormal object.

[0191] In a separate embodiment, the integrated electronic device (100a) can receive and analyze images from a plurality of cameras connected to the same network. This is explained based on FIG. 12.

[0192] Referring to FIG. 12, the integrated electronic device (100a) can be connected to the same network as multiple cameras (300-1 to 300-n) located inside the same building. At this time, the multiple cameras (300-1 to 300-n) can be connected via wired communication (e.g., Ethernet, USB, etc.) and can also be connected via wireless communication (e.g., Wi-Fi, Bluetooth, Zigbee, etc.). Additionally, unlike the electronic device (100) or the integrated electronic device (100a), the multiple cameras (300-1 to 300-n) may be cameras that perform only shooting functions without including an artificial intelligence processor (102, 102a).

[0193] Each of the connected multiple cameras (300-1 to 300-n) can transmit video to an integrated electronic device (100a) via wired or wireless communication. The integrated electronic device (100a) can analyze the received video using an artificial intelligence processor (102a). The integrated electronic device (100a) analyzes the received video, and if an abnormal object is detected, converts the private IP address to a public IP address via NAT (Network Address Translation) and transmits the fact of detection of the abnormal object and information about the detected object to the central control center (200) via the Internet (10). Specific details regarding this are as described above in FIG. 10. As a separate embodiment, the central control center (200) may also directly transmit a notification to a user terminal device (1200). Details regarding this are explained based on FIG. 13.

[0194] Referring to FIG. 13, the user terminal device (1200) may receive a signal from the central control center (200) requesting that a notification be displayed. Based on the received signal, the user terminal device (1200) may display a warning notification such as "abnormal object detection".

[0195] At this time, the interface element (1210) displaying the warning notification may display a button (1211) that allows real-time camera video to be viewed. When input for the button (1211) is received, the user terminal device (1200) may transmit a signal requesting the electronic device (100) to stream the camera video. Based on the request, the electronic device (100) may transmit the camera video to the user terminal device (1200) in real time using port forwarding.

[0196] The types of user terminal devices and internal devices are not limited to the aforementioned types and may include various electronic devices.

[0197] In addition, in the example described above, the electronic device (100) is described as being a combination of a camera (101) and an artificial intelligence processor (102), but the configuration of the electronic device (100) may not be limited thereto and may be composed of a combination of various devices. Furthermore, it is obvious that not only the electronic device (100) but also the integrated electronic device (100a) can perform the operations of FIGS. 1 to 13.

[0198] Although various embodiments have been described above, each embodiment is not necessarily implemented individually, and may be combined with at least one other embodiment, either wholly or partially, to be implemented together in a single product.

[0199] Meanwhile, embodiments of the present disclosure may also be implemented in the form of a recording medium containing computer-executable instructions, such as program modules executed by a computer. A computer-readable medium may be any available medium accessible by a computer and includes both volatile and non-volatile media, and both removable and non-removable media. Additionally, a computer-readable medium may include computer storage media and communication media. Computer storage media include both volatile and non-volatile, removable and non-removable media implemented by any method or technique for storing information, such as computer-readable instructions, data structures, program modules, or other data. Communication media may typically include other data of modulated data signals, such as computer-readable instructions, data structures, or program modules.

[0200] Additionally, computer-readable storage media may be provided in the form of non-transitory storage media. Here, 'non-transitory storage media' simply means that it is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily. For example, 'non-transitory storage media' may include a buffer in which data is stored temporarily.

[0201] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., downloadable app) may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0202] The foregoing description of the present disclosure is for illustrative purposes only, and those skilled in the art will understand that other specific forms can be easily modified without altering the technical spirit or essential features of the present disclosure. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form.

[0203] The scope of the present disclosure is defined by the claims set forth below rather than by the detailed description above, and all modifications or variations derived from the meaning and scope of the claims and equivalent concepts thereof should be interpreted as being included within the scope of the present disclosure.

Claims

1. In an electronic device, A camera including a communication circuit for communicating with an external device; and It includes an artificial intelligence processor configured to be connected to an intranet through the above communication circuit and to execute an image analysis module that analyzes images using an artificial intelligence model; The above camera is, An electronic device configured such that when the image analysis module detects an event based on an image acquired through the camera, it transmits information about the event to the external device and receives a control signal from the external device.

2. In Paragraph 1, The above event is, An electronic device comprising a case in which the artificial intelligence processor identifies an object detected from the image as not being a person or object under preset conditions.

3. In Paragraph 1, The above electronic device is, Using the above camera, the video is transmitted in real time to the above AI processor via RTSP (Real Time Streaming Protocol) streaming based on the IP address of the above AI processor, and An electronic device that detects the event based on the real-time transmitted video using the artificial intelligence processor.

4. In Paragraph 1, The electronic device is configured to transmit event information to the external device using the camera as a network gateway.

5. In Paragraph 4, The above electronic device is, An electronic device that performs Network Address Translation (NAT) and transmits the detected event information through the camera to the external device.

6. In Paragraph 4, The above electronic device is, An electronic device that, upon receiving a signal from the external device commanding the provision of a notification to a user in accordance with the occurrence of the event, performs port forwarding through the camera and transmits the signal to an internal device connected to the same network as the electronic device.

7. In Paragraph 1, An electronic device configured such that the image analysis module and the camera are connected by a first cable capable of TCP / IP protocol and a second cable which is a serial cable separate from the first cable, and the image is transmitted through the first cable and at least some control data is transmitted through the second cable.

8. In Paragraph 1, The above image is an electronic device comprising dynamic and still images acquired through the camera.

9. A method for controlling an electronic device including a camera and an artificial intelligence processor, A step of transmitting via an intranet to an AI processor configured to execute an image analysis module that analyzes the image using an AI model, the image acquired through the above camera; When the artificial intelligence processor detects an event based on the image, the step of transmitting information about the event to an external device; and The method includes the step of receiving a control signal from the external device. A control method comprising a camera having a communication circuit for communicating with an external device.

10. In Paragraph 9, The above event is, A control method comprising a case in which the artificial intelligence processor identifies an object detected from the image as not being a person or object of a preset condition.

11. In Paragraph 9, The step of transmitting the above-mentioned acquired image to the artificial intelligence processor is, A step of transmitting the video to the AI ​​processor in real time via RTSP (Real Time Streaming Protocol) streaming based on the IP address of the AI ​​processor; and A control method comprising the step of detecting the event based on the real-time transmitted video using the artificial intelligence processor.

12. In Paragraph 9, A control method in which the electronic device is configured to use the camera as a network gateway to transmit the event information to the external device.

13. In Paragraph 12, The step of transmitting information about the above event to the server is, A control method comprising the step of performing Network Address Translation (NAT) to transmit the detected event information to the external device.

14. In Paragraph 12, The step of receiving a control signal from the above external device is, A control method comprising the step of: receiving a signal from the external device commanding the provision of a notification to a user upon the occurrence of the event, performing port forwarding to transmit the signal to an internal device connected to the same network as the electronic device.

15. In Paragraph 9, A control method in which the artificial intelligence processor and the camera are connected by a first cable capable of TCP / IP protocol and a second cable which is a serial cable separate from the first cable, wherein the image is transmitted through the first cable and at least some control data is transmitted through the second cable.

16. In Paragraph 9, The above image is a control method comprising dynamic and static images acquired through the camera.

17. A non-transient computer-readable recording medium storing one or more instructions executed by a processor of said electronic device to enable said electronic device, including a camera and an artificial intelligence processor, to perform an operation, wherein said operation is, A step of transmitting via an intranet to an AI processor configured to execute an image analysis module that analyzes the image using an AI model, the image acquired through the above camera; When the artificial intelligence processor detects an event based on the image, the step of transmitting information about the event to an external device; and The method includes the step of receiving a control signal from the external device; The above camera is a recording medium comprising a communication circuit for communicating with the external device.

18. In Paragraph 17, The above electronic device is a recording medium configured to transmit the event information to a server using the camera as a network gateway.

19. In Paragraph 18, The step of transmitting information about the above event to the server is, A recording medium comprising the step of performing Network Address Translation (NAT) to transmit the detected event information to the external device.

20. In Paragraph 18, The step of receiving a control signal from the above external device is, A recording medium comprising the step of: receiving a signal from the external device commanding the provision of a notification to a user upon the occurrence of the event, performing port forwarding to transmit the signal to an internal device connected to the same network as the electronic device.

Images