Method and system for document understanding

Abstract

The present invention relates to a method and a system for document understanding. The document understanding method according to the present invention may comprise the steps of: specifying at least one document to be analyzed; inputting the at least one document into a document analysis model trained on the basis of a training document data set including at least one augmented document data; by analyzing the at least one document using the document analysis model, identifying at least one layout area having an attribute different from that of another layout area in the at least one document; extracting at least one content included in each of the identified at least one layout area; generating output data for the at least one document by using the extracted at least one content; and providing the output data to a user terminal.

Description

Document Understanding Methods and Systems

[0001] The present invention relates to a document understanding method and system, and provides a document understanding method and system using Deep Document Understanding (DDU) technology.

[0002] Documents are recorded through characters or symbols (or codes) to preserve and transmit thoughts, ideas, intentions, and information; in a broad sense, they can include everything that contains meaning, such as pictures, photographs, and videos. This means that information is recorded and preserved in various forms and can serve as a means of communication.

[0003] In this regard, a large volume of electronic documents created via computers is being utilized in various fields in modern society. These documents contain not only text but also various forms of information, such as tables, images, and graphs. For example, documents used in industrial settings, such as papers, patents, technical reports, and manuals, include not only text but also diverse forms of information like tables, graphs, diagrams, images, mathematical formulas, and molecular structural formulas.

[0004] While humans can read and understand the various forms of information contained in such documents, computers cannot fully comprehend complex information in the same way; therefore, analyzing and utilizing document content requires the cumbersome task of converting it into a format recognizable by computers. However, as documents contain a mixture of diverse information, manually reviewing or analyzing them can consume a significant amount of time and cost.

[0005] To address these issues, various artificial intelligence models have been proposed to automatically understand and analyze the content of documents. In particular, Document Layout Analysis (DLA) AI models, which analyze the layout within a document to distinguish components such as text, images, and tables and identify the relationships between them, are utilized as a core element of document understanding technology. These document analysis models are trained to identify multiple layout areas within a document and predict the location, size, type, and reading order of each area, thereby enabling the structural interpretation of the information contained in the document.

[0006] However, training such document layout analysis AI models requires large-scale training data regarding layout areas. Conventional methods for generating training data have largely involved either completely synthesizing documents or attaching labels to actual documents.

[0007] The existing fully synthetic method artificially generates documents using a computer. It has the advantages of rapidly generating large amounts of data, automatically creating accurate annotations of various granularities at the character, word, and sentence levels, and having relatively low data generation costs. However, this method has drawbacks, such as difficulty in sufficiently mimicking the complex layout structures found in actual documents, a lack of realism in the generated documents, and limited layout diversity.

[0008] On the other hand, the existing method of attaching labels to actual documents has the advantage of accurately reflecting the layout and visual characteristics of the actual documents, allowing for the inclusion of various layout forms, and naturally incorporating document deformations or noise that occur in real-world environments. However, this method requires manual labeling of layout areas, resulting in high costs and time, and limits the ability to scale the dataset.

[0009] As such, conventional methods for generating training data for document analysis models have limitations that conflict with each other in terms of data realism, diversity, generation cost, and scalability. Consequently, there is a continuously increasing demand for technologies capable of generating large-scale training data by efficiently augmenting various forms of data while maintaining the layout structure of actual documents and minimizing manual human intervention.

[0010] Furthermore, there is a need for technology that utilizes document understanding AI models trained with such augmented learning data to more deeply understand various forms of information contained in documents, and based on this, provides document analysis and utilization services optimized for various fields.

[0011] The present invention is intended to provide a document understanding method and system capable of effectively understanding various forms of documents.

[0012] More specifically, the present invention is intended to provide a document understanding method and system that can identify a layout area constituting a document and effectively understand complex forms of content included in the layout area.

[0013] Furthermore, the present invention aims to provide a document understanding method system capable of deeply understanding content having various data characteristics included in a document by augmenting training data for training a document analysis model and using a document analysis model trained with a training document data set including the augmented training data.

[0014] To solve the problem described above, a document understanding method according to the present invention, performed by a computer, may include: a step of specifying at least one document to be analyzed; a step of inputting said at least one document into a document analysis model trained based on a training document data set including at least one augmented document data; a step of analyzing said at least one document in said document analysis model to identify at least one layout area having different attributes from said at least one document; a step of extracting at least one content included in each of said at least one layout area; a step of generating output data for said at least one document using said at least one content; and a step of providing said output data to a user terminal.

[0015] In an embodiment, the document analysis model is characterized by being a model trained using a training document dataset that includes augmented document data generated based on at least one training document data.

[0016] In an embodiment, the document analysis model is characterized by identifying the layout area containing at least one content from the document through analysis of the document, and generating layout area data corresponding to the document.

[0017] In an embodiment, the layout area data is characterized by including at least one of a label, location information, size information, and a score corresponding to each of the contents included in the document.

[0018] In an embodiment, the step of extracting the content is characterized by processing the layout area data and the document as inputs for a plurality of models corresponding to the data characteristics of each of the content, and extracting the content data included in the layout area based on the output of the plurality of models.

[0019] In an embodiment, the step of generating the output data is characterized by performing multimodal data processing on content data output from the plurality of models based on the layout area data to generate a structured document understanding result, and arranging the structured document understanding result to generate the output data.

[0020] In an embodiment, the multimodal data processing is characterized by including a data processing method that combines text-based processing and non-text-based processing.

[0021] In an embodiment, the structured document understanding result is characterized by being generated to reflect the sequential or hierarchical relationship between the contents.

[0022] In an embodiment, the plurality of models is characterized by including models corresponding to different content types.

[0023] In an embodiment, the step of extracting the content data is characterized by including different preprocessing or postprocessing data processing depending on the data characteristics of the content.

[0024] A document understanding method characterized in that, in an embodiment, the layout area data further includes structural information indicating an inclusion relationship or adjacency relationship between the layout areas.

[0025] In an embodiment, the layout area data is characterized by further including score information related to the reliability or accuracy of the document analysis model.

[0026] In an embodiment, the content comprises at least one of text, a table, a chart, a molecular structural formula, or chemical reaction information, and each of the content is included in a layout area corresponding to the data characteristics of each of the content. In an embodiment, the method includes the step of generating at least one augmented document data using at least one training document data, and the step of generating the augmented document data may include the step of specifying the at least one training document data, the step of deactivating specific content included in at least one specific layout area constituting the at least one training document data, and the step of generating the augmented document data including alternative content having the same data characteristics as the specific content in the deactivated at least one specific layout area.

[0027] In the embodiment, in the deactivating step, the specific content can be deactivated based on the layout structure of each of the at least one specific layout area, based on annotation data corresponding to the at least one specific layout area.

[0028] In an embodiment, the deactivating step may include the step of extracting background color information for each of the at least one specific layout area using the annotation data, and the step of deactivating the specific content using the background color information so that the specific content area corresponding to the specific content corresponds to the background color of the at least one specific layout area.

[0029] In an embodiment, the step of generating the augmented document data may include: using the annotation data to identify at least one alternative content having the same data characteristics as the specific content included in each of the at least one specific layout areas among the plurality of candidate content data; reflecting at least one alternative content in each of the at least one specific layout areas based on the layout structure of each of the at least one specific layout areas; and generating the augmented document data in which at least one alternative content is reflected in each of the at least one specific layout areas.

[0030] In an embodiment, in the step of reflecting the alternative content, at least one alternative content may be reflected in each of the at least one specific layout area based on the inclusion relationship of each of the at least one specific layout area.

[0031] In an embodiment, the type of the at least one specific layout area is determined based on the data characteristics of the specific content included in each of the at least one specific layout areas, and if any one of the at least one specific layout areas has a first type layout area that includes at least one second type layout area, the at least one second type layout area included in any one of the first type layout areas can be disabled.

[0032] In an embodiment, the step of generating the augmented document data including the alternative content may determine the content color corresponding to the alternative content by using the background color information for each of the at least one specific layout area.

[0033] In an embodiment, the content color may be determined to have a color difference from the background color according to the background color information based on a predetermined specific threshold.

[0034] In an embodiment, the step of generating the augmented document data including the alternative content may generate the augmented document data by reflecting at least one noise effect on the alternative content based on a preset probability.

[0035] In an embodiment, the annotation data may include at least one of position data, size data, category data, and reading order data corresponding to each of the at least one specific layout area.

[0036] In an embodiment, the document analysis model can identify the at least one layout area containing the at least one content from the at least one document through analysis of the at least one document, and output layout area data corresponding to the at least one document.

[0037] In an embodiment, in the step of extracting the at least one content, the layout area data and the at least one document are processed as inputs to a plurality of models corresponding to the data characteristics of each of the at least one content, so that the at least one content included in the at least one layout area can be extracted from the plurality of models.

[0038] In an embodiment, the step of generating the output data may include: performing multimodal data processing for the at least one content output from the plurality of models based on the layout area data to generate a structured document understanding result related to the at least one content; and arranging the structured document understanding result according to a preset order to generate the output data.

[0039] In an embodiment, the at least one content includes at least one of text, a table, a chart, a molecular structure, and chemical reaction information, and may be included in a respective layout area corresponding to the data characteristics of each of the at least one content.

[0040] In an embodiment, the layout area data may include at least one of a label corresponding to each of the at least one content included in the at least one document, location information for each of the at least one content, and a score for each of the at least one content.

[0041] Meanwhile, a document understanding system according to the present invention includes a memory for storing instructions and at least one processor electrically connected to the memory, and when the instructions are executed by the at least one processor, the at least one processor identifies at least one document to be analyzed and inputs the at least one document into a document analysis model learned based on a learning document data set including at least one augmented document data, and in the document analysis model, analyzes the at least one document to identify at least one layout area having different attributes from the at least one document, extracts at least one content included in each of the identified at least one layout area, and generates output data for the at least one document using the extracted at least one content; and can provide the output data to a user terminal.

[0042] A program that is executed by one or more processes in an electronic device and can be stored on a computer-readable recording medium, wherein the program may include instructions for performing the steps of: specifying at least one document to be analyzed; inputting said at least one document into a document analysis model learned based on a learning document data set containing at least one augmented document data; analyzing said at least one document in said document analysis model to identify at least one layout area having different attributes from said at least one document; extracting at least one content included in each of said at least one layout area; generating output data for said at least one document using said extracted at least one content; and providing said output data to a user terminal.

[0043] A method for training a document analysis model according to the present invention may include the steps of: specifying at least one training document data; deactivating specific content included in at least one specific layout area constituting the at least one training document data; generating augmented document data including alternative content having the same data characteristics as the specific content in the deactivated at least one specific layout area; and training a document analysis model using a training document data set including the augmented document data.

[0044] In an embodiment, the step of deactivating the specific content is characterized by being performed based on the layout structure of each of the at least one specific layout area, based on annotation data corresponding to the at least one specific layout area.

[0045] In an embodiment, the step of deactivating the specific content may include the step of extracting background color information for each of the at least one specific layout area using annotation data, and the step of deactivating the specific content using the background color information so that the specific content area corresponding to the specific content corresponds to the background color of the specific layout area.

[0046] In an embodiment, the step of generating augmented document data comprises: a step of identifying, among a plurality of candidate content data using annotation data, alternative content having the same data characteristics as specific content included in each of the specific layout areas; a step of reflecting at least one alternative content in each of the specific layout areas based on the layout structure; and a step of generating augmented document data in which at least one alternative content is reflected.

[0047] In an embodiment, the step of reflecting the alternative content is characterized by being performed based on the inclusion relationship of each of the at least one specific layout area.

[0048] In an embodiment, the type of at least one specific layout area is determined based on the data characteristics of specific content included in each of the specific layout areas, and if any one of the specific layout areas includes a first type layout area, the second type layout area included in the first type layout area is disabled.

[0049] In an embodiment, the step of generating the augmented document data is characterized by determining the content color corresponding to the alternative content using background color information for each of the specific layout areas.

[0050] In an embodiment, the content color is characterized by being determined to have a color difference from the background color according to the background color information based on a predefined threshold value.

[0051] In an embodiment, the step of generating the augmented document data is characterized by being performed by reflecting at least one noise effect in the replacement content based on a preset probability.

[0052] In an embodiment, the annotation data is characterized by including at least one of position data, size data, category data, and read order data corresponding to each of the specific layout areas.

[0053] As described above, the document understanding method and system according to the present invention can increase (or enhance) user convenience by recognizing the structure of various types of documents and the relationships between multiple contents (e.g., text, tables, graphs, charts, molecular structural formulas, etc.) included in the documents, thereby automatically extracting accurate information. Through this, users can minimize the time required to find necessary information in documents and reduce inconvenience.

[0054] Furthermore, according to the document understanding method and system of the present invention, a plurality of contents included in a document can be identified individually, and a plurality of contents can be automatically extracted from the document by using a plurality of models specialized for each of the plurality of contents having different data characteristics. Through this, the user can receive the necessary information quickly and accurately.

[0055] Furthermore, the document understanding method and system according to the present invention can prevent information omission and enhance the sophistication of analysis by deeply analyzing visual information of various forms of content included in a document. Through this, the present invention can accurately process a vast amount of documents, thereby improving the user's overall work or business processing speed.

[0056] Furthermore, according to the document understanding method and system of the present invention, by deeply analyzing various forms of content included in a document and providing a solution optimized for the user based on this, the efficiency of document processing can be maximized and the user's overall work efficiency can be improved.

[0057] Furthermore, according to the document understanding method and system of the present invention, it is possible to understand text, graphs, and tables contained in a document, while simultaneously interpreting complex molecular structural formulas and chemical reaction information. Through this, the present invention can enhance the research efficiency of chemists and support innovative discoveries in the field of chemistry. In other words, the present invention can satisfy the needs of the research field through a customized model that understands even information related to the chemical domain within the document.

[0058] Furthermore, according to the document understanding method and system of the present invention, molecular structural formulas can be accurately recognized in documents using a model specialized for molecular structural formulas, and a large-scale database of molecular structural formulas can be constructed based on this. Through this, researchers can utilize the database to search for chemical structural formulas and efficiently carry out large-scale analysis tasks required for research, thereby reducing the time required for research and / or development.

[0059] Furthermore, according to the document understanding method and system of the present invention, a plurality of contents extracted from a document using a plurality of models and output data generated based on said plurality of contents can be visualized and provided through a user interface. Through this, the user can intuitively recognize the information needed and understand it more quickly, thereby improving the efficiency of work and / or research.

[0060] Furthermore, according to the document understanding method and system of the present invention, augmented document data having the same layout structure as the actual training document data used for training a Document Layout Analysis (DLA) model can be generated. Through this, large-scale training data reflecting various layout characteristics and visual variations of actual documents can be efficiently secured, and the generalization performance and document understanding accuracy of the trained document analysis model can be effectively improved.

[0061] FIG. 1 illustrates an example of a block diagram of a computing system in which the present invention can be implemented.

[0062] FIG. 2 illustrates an example of a block diagram of a computing device that may be included in a user computing device, a server computing system, and a training computing system, as an embodiment of a computing system in which the present invention can be implemented.

[0063] FIG. 3 illustrates an example of a block diagram in another aspect of a computing device, which is one of the components of a computing system according to the present invention.

[0064] FIGS. 4, FIGS. 5a, and FIGS. 5b are conceptual diagrams for explaining a document understanding system according to the present invention.

[0065] FIG. 6 is a flowchart illustrating a method for understanding documents according to the present invention.

[0066] FIGS. 7, FIGS. 8a, and FIGS. 8b are flowcharts illustrating a method for understanding documents according to the present invention.

[0067] FIGS. 9 and FIGS. 10a to 10f are flowcharts and conceptual diagrams for explaining the process of generating augmented document data according to the present invention.

[0068] FIGS. 11a to 11e are conceptual diagrams for explaining the process of generating output data according to the present invention.

[0069] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Identical or similar components are assigned the same reference number regardless of the drawing symbols, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably solely for the ease of drafting the specification and do not have distinct meanings or roles in themselves. Furthermore, in describing the embodiments disclosed in this specification, if it is determined that a detailed description of related prior art could obscure the essence of the embodiments disclosed in this specification, such detailed description will be omitted. Additionally, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification; the technical concept disclosed in this specification is not limited by the attached drawings, and it should be understood that they include all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the present invention.

[0070] Terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. These terms are used solely for the purpose of distinguishing one component from another.

[0071] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0072] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0073] In this application, terms such as “comprising” or “having” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0074] Hereinafter, the present invention will be examined in more detail with reference to the attached drawings. FIG. 1 illustrates an example of a block diagram of a computing system in which the present invention can be implemented. FIG. 2 illustrates an example of a block diagram of a computing device that may be included in a user computing device, a server computing system, and a training computing system as an embodiment of a computing system in which the present invention can be implemented. FIG. 3 illustrates an example of a block diagram of a computing device in another aspect that is one of the components of a computing system.

[0075] Meanwhile, FIG. 1 illustrates an example of a block diagram of a computing system in which the present invention can be implemented. In this regard, the document understanding system according to the present invention can be implemented through a computing device described below and can perform data processing related to the document understanding method described above.

[0076] Referring to FIG. 1, a method for automatically converting a plurality of contents (or at least one content) extracted from various forms of documents according to one embodiment of the present invention into data and providing various services and / or functions based thereon, and a computing system (10000) for performing a document understanding method according to the present invention may include at least one computing device. At this time, the at least one computing device may be a single processor or a multi-processor computing device.

[0077] The components of at least one computing device of the present invention may include at least one processor, memory, other hardware, and various hardware components such as a system bus (not shown) that connects various system components so that they can transmit and receive data to and from each other (e.g., telecommutatively connected, physically connected, electrically connected), and the components of at least one computing device are not limited thereto and may be very diverse.

[0078] Meanwhile, at least one computing device included in a computing system (10000) that performs a method for automatically converting multiple contents extracted from various types of documents into data and providing various services and / or functions based thereon, and a document understanding method according to the present invention, may be connected to communicate via a network (1070). For example, at least one computing device included in the computing system (10000) may be clustered or may be part of a local area network (LAN). Additionally, at least one computing device may be part of a wide area network (WAN) or connected to at least one of a client-server network and a peer-to-peer network within the cloud.

[0079] Meanwhile, when at least one computing device is used in at least one of a network environment and a cloud computing environment, the at least one computing device may be connected to at least one of a public and private network through a network interface or adapter. In one embodiment, other communication connection devices, such as a modem, may be used to establish communication through the network. The modem may be at least one of an internal modem and an external modem, and may be connected to a system bus through a network interface or a specific mechanism, etc. A wireless network component consisting of an interface and an antenna may be coupled to the network through a device such as an access point, a peer computer, etc. In the present invention, the method of connecting at least one computing device to communicate through the network (1070) is not limited, and it may be connected to communicate in a manner different from the described example.

[0080] Furthermore, other computer-type devices and / or systems not shown in FIG. 1 may also interact technically with at least one computing device or other system through one or more connections to the network (1070) via a network interface. Here, the network interface may include network interface equipment such as a physical network interface controller (NIC) or a virtual network interface (VIF).

[0081] The network (1070) of the present invention may include various forms such as the Internet, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), 5G (5th Generation Mobile Telecommunication), Bluetooth (Bluetooth™ Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra-Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi Direct, Wireless USB (Wireless Universal Serial Bus), etc., and in the present invention, data transmission may be performed based on standard communication protocols such as TCP / IP, HTTP, SSL, etc.

[0082] A method for automatically converting a plurality of contents extracted from various types of documents according to the present invention into data and providing various services and / or functions based thereon, and a computing system (10000) for performing a document understanding method according to the present invention may include at least one of a user computing device (1010), a training computing system (1050), and a server computing system (1030).

[0083] A user computing device (1010) according to the present invention may be understood as a computing device comprising at least one processor (1011) and a memory (1012) that perform a method of automatically converting a plurality of contents extracted from various types of documents into data and providing various services and / or functions based thereon, and a method of understanding documents according to the present invention. For example, the user computing device (1010) may include at least one computing device among a smartphone, a smart TV, a laptop computer, a desktop computer, a digital broadcasting terminal, a PDA (personal digital assistants), a PMP (portable multimedia player), a navigation device, a slate PC, a tablet PC, an ultrabook, a wearable device (e.g., a smartwatch, a smart glass, and a head-mounted display).

[0084] At least one processor (1011) constituting the user computing device (1010) may include one or more general-purpose processors and / or one or more special-purpose processors. For example, at least one and / or at least one processor (1011) constituting the user computing device (1010) may be composed of at least one of a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a neural network processing unit (NPU), an arithmetic logic unit (ALU), a floating-point arithmetic unit (FPU), an application integrated circuit, an application semiconductor (ASIC), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, and / or electrical units for performing other functions, or a plurality of electrically connected processors.

[0085] Furthermore, at least one and / or at least one processor (1011) may be configured to execute computer-readable instructions contained in memory (1012) and / or other instructions described herein.

[0086] The memory (1012) constituting the user computing device (1010) according to the present invention may include volatile memory, non-volatile memory, fixed media, removable media, magnetic media, optical media, semiconductor media and / or other types of physically durable storage media.

[0087] For example, the memory (1012) may include one or more non-transient / transient computer-readable storage media such as RAM, ROM, HDD (Hard Disk Drive), SSD (Solid State Disk), SSD (Silicon Disk Drive), EEPROM, EPROM, flash memory device, magnetic disk, etc., and may include web storage of a server that performs the storage function of memory on the internet. This memory (1012) may store data and instructions necessary for the operation of an application to automatically convert multiple contents extracted from various types of documents by the at least one and / or at least one processor (1011) into data and to provide various services and / or functions based thereon.

[0088] A user computing device (1010) may include one or more user input components (1021) that detect user input. For example, the user input component (1021) may also be referred to as a user interface module. The user input component (1021) may include a touch screen, a computer mouse, a keyboard, a keypad, a touchpad, a trackball, a joystick, a voice recognition module, or other similar devices. However, the present invention does not limit the type of user input component (1021). In this case, the user input component (1021) in the present invention does not necessarily mean a hardware means, but can be understood as a channel for receiving input from a user. Meanwhile, the user of the present invention may refer to an automated agent, script, playback software, etc., that operates on behalf of one or more people.

[0089] A user can interact with a computing system (10000) including at least one computing device through input text, touch, voice, movement, computer vision, gestures and / or other forms of input / output using a user input component (1021). For example, the user input component (1021) may include one or more of a command line interface (CLI), a graphical user interface (GUI), a natural user interface (NUI), a voice command interface and / or other user interface (UI) representations.

[0090] Between the user input component (1021) and the user computing device (1010), one or more application programming interface (API) calls may be made based on user input received from the user interface and / or network.

[0091] Here, the expression "based on" may be interpreted to include cases where it is based on the use of a specific configuration, modified from, derived from, influenced by, dependent on, or otherwise derived from a specific configuration. In some embodiments, an API call may be configured for a specific API, which may be interpreted or converted into an API call configured for another API. Here, an API may refer to a defined interface or connection between computers or between computer programs.

[0092] In one embodiment, the user computing device (1010) may store at least one machine learning model (1020). For example, the user computing device (1010) may be various machine learning models, such as a plurality of neural networks (e.g., deep neural networks) that automatically convert a plurality of contents extracted from various types of documents into data and provide various services and / or functions based thereon, and other types of machine learning models including non-linear models and / or linear models, and may be composed of a combination thereof.

[0093] According to an embodiment of the present invention, a user computing device (1010) may use a local or / and external machine learning model (1020) to automatically convert multiple contents extracted from various types of documents into data and provide various services and / or functions based thereon, and may perform a document understanding method according to the present invention. Alternatively, the user computing device (1010) may use a machine learning model (1040) provided by a server to automatically convert multiple contents extracted from various types of documents into data and provide various services and / or functions based thereon, and may perform a document understanding method according to the present invention.

[0094] In addition, according to another embodiment of the present invention, a server computing system (1030) communicating with a user computing device (1010) can provide various services and / or functions based on a plurality of content extracted from various types of documents to the user computing device (1010) on an application or / and the web, in accordance with a request from a user received through the user computing device (1010).

[0095] In addition, according to another embodiment of the present invention, at least a part of a user computing device (1010) and a server computing system (1030) are interconnected to perform a process of automatically converting multiple contents extracted from various types of documents into data, and based on this, various services and / or functions can be provided to the user.

[0096] Additionally, according to various embodiments of the present invention, a user computing device (1010) and / or a server computing system (1030) can learn a machine learning model (1020, 1040) performed in a method for automatically converting multiple contents extracted from various types of documents into data and providing various services and / or functions based thereon, and a document understanding method according to the present invention, through interaction with a training computing system (1050) that is communicatedly connected via a network (1070). At this time, the training computing system (1050) may be a computing system separate from the server computing system (1030). Alternatively, in some embodiments, the training computing system (1050) may be a part of the server computing system (1030) or a part of the user computing device (1010).

[0097] Meanwhile, the server computing system (1030) may include at least one processor (1031) and memory (1032). Here, the processor (1031) may be composed of at least one or a plurality of electrically connected processors among a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a neural network processing unit (NPU), an application integrated circuit, an application semiconductor (ASIC), an arithmetic logic unit (ALU), a floating-point arithmetic unit (FPU), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, and / or other electrical units for performing functions. For example, at least one processor (1031) may include a circuit and a transistor configured to execute instructions from memory (1032).

[0098] The memory (1032) constituting the server computing system (1030) according to the present invention may include volatile memory, non-volatile memory, fixed media, removable media, magnetic media, optical media, semiconductor media, and / or other types of physically durable storage media. For example, the memory (1032) may include one or more non-transient / transient computer-readable storage media such as RAM, ROM, HDD (Hard Disk Drive), SSD (Solid State Disk), SSD (Silicon Disk Drive), EEPROM, EPROM, flash memory device, magnetic disk, etc., and combinations thereof, and may include web storage of a server that performs the storage function of memory over the internet. Additionally, the server computing system (1030) may further include a data storage (data store). For example, the data storage may be composed of at least one of a relational database, a NoSQL database, a data warehouse, and a local file system.

[0099] In the memory (1032) constituting the server computing system (1030) according to the present invention, data and instructions necessary for the operation of an application to automatically convert a plurality of contents extracted from various types of documents by the at least one processor (1031) into data and to provide various services and / or functions based thereon may be stored.

[0100] In one embodiment, the server computing system (1030) may be composed of a single device or a plurality of computing devices, and may be configured to operate according to a sequential or parallel computing architecture. Additionally, a distributed processing system may be configured with a plurality of networked devices.

[0101] Meanwhile, the training computing system (1050) may include at least one processor (1051) and memory (1052). The model trainer (1060) is a logical component that executes the training of at least one machine learning model (1020, 1040) and may be implemented in the form of hardware, firmware, or software. For example, the model trainer (1060) may be executed by the processor (1051) after loading training data (1061) stored in a storage device into memory (1052). For example, the model trainer (1060) may be configured to execute one or more operations (e.g., model training, model reconstruction, model validation, model testing) on ​​at least one machine learning model.

[0102] The machine learning model of the present invention may include at least one of a statistical model, an algorithm, a neural network (NN), a convolutional neural network (CNN), a generative neural network (GNN), a Word2Vec model, a Bag of Words model, a TF-IDF (document frequency-inverse document frequency) model, a GPT (Generative Pre-trained Transformer) model (or other autoregressive models), a PPO (Proximal Policy Optimization) model, a nearest neighbor model (e.g., a k-nearest neighbor model), a linear regression model, a K-means clustering model, a Q-learning model, a TD (Temporal Difference) model, a Deep Adversarial Network model, and all other types of models further described herein.

[0103] Specifically, the model trainer (1060) may execute operations to train a machine learning model, and said operations may include at least one of adding, removing, and modifying model parameters. At this time, the training of the machine learning model may be at least one of supervised learning, semi-supervised learning, and unsupervised learning. In one embodiment, the training of the machine learning model may include the step of repeatedly inputting training data (1061) based on epochs and repeatedly performing the machine learning model training process configured in this way. Here, an epoch may refer to a unit in which the entire set of training data (1061) undergoes forward and backpropagation processing once. In some implementations, different levels of training methods (e.g., supervised learning, semi-supervised learning, unsupervised learning) may be used for different epochs.

[0104] The training data (1061) of the present invention may include input data and / or data previously output from at least one machine learning model (e.g., recursive learning feedback).

[0105] At least one parameter of a machine learning model may include at least one of a seed value, a model node, a model layer, an algorithm, a function, connections between different machine learning models, connections between parameters, machine learning model constraints, and other digital components that influence the output of the machine learning model. In this case, model connections between different machine learning models may include or represent relationships between model parameters and / or models, which may be dependent or interdependent, hierarchical, and / or static or dynamic. The combinations and configurations of model parameters described herein may be too complex to be maintained or utilized by human cognitive abilities.

[0106] In the present invention, the machine learning parameters described according to the embodiments are not limited, and a single machine learning model may further include a plurality of model parameters.

[0107] Meanwhile, FIG. 2 illustrates an example of a block diagram of a computing device (1100) that may be included in a user computing device (1010), a server computing system (1030), and a training computing system (1050), as an embodiment of a computing system (10000) in which the present invention can be implemented.

[0108] As illustrated in FIG. 2, the computing device (1100) may include at least one application (e.g., Application 1 to Application N), and each of the at least one application may include a machine learning library and a model execution environment for performing a method of automatically converting a plurality of contents extracted from various forms of documents based on machine learning into data and providing various services and / or functions based thereon, and a document understanding method according to the present invention. The at least one application included in the computing device (1100) may communicate with the sensor, context manager, device state manager, or additional component(s) within the computing device (1100) via an API (Application Programming Interface). In one embodiment, the at least one application may perform an interface with device components, such as receiving sensor data or state data or transmitting prediction results to an output device via a public or private API.

[0109] Meanwhile, FIG. 3 illustrates an example of a block diagram in another aspect of a computing device (1200) which is one of the components of a computing system (10000) that performs a method for automatically converting a plurality of contents extracted from various forms of documents according to an embodiment of the present invention into data and providing various services and / or functions based thereon, and a method for understanding documents according to the present invention.

[0110] A computing device (1200) according to the present invention may include at least one application (e.g., Application 1 to Application N), and at least one application may communicate with a central intelligence layer (1210). Each application may interact with a shared model within the central intelligence layer (1210) through an API (e.g., a common API).

[0111] The central intelligence layer (1210) includes one or more machine learning models and may share them among multiple applications or provide them independently to each. In one embodiment, the central intelligence layer (1210) may be integrated as part of an operating system or implemented as a separate logical layer.

[0112] Additionally, the central intelligence layer (1210) can communicate with the central device data layer (1220). The central device data layer (1220) can integrate and store various types of documents containing multiple contents stored within the computing device (1200), automatically convert multiple contents from the documents into data, and provide this as input data necessary to provide various services and / or functions based on this. Each device component (e.g., sensor, state manager, etc.) can communicate with the central device data layer (1220) through a private API, etc.

[0113] The technology described in this specification may be composed of a single or multiple computing devices, and a method for automatically converting multiple contents extracted from various types of documents into data and providing various services and / or functions based thereon, and a machine learning model for performing the document understanding method according to the present invention may be executed sequentially or in parallel on a single component or multiple distributed components. The data storage, machine learning model, and application may be configured and operated in a distributed manner locally or over a network, and these configurations can be flexibly applied to various system architectures.

[0114] The above describes a computing system in which the document understanding method according to the present invention can be implemented. Below, the document understanding method and system according to the present invention will be described in general.

[0115] Meanwhile, the present invention relates to a document understanding method and system capable of effectively understanding various types of documents. Specifically, the present invention can provide a service and / or function that automatically converts multimodal information (e.g., layout, text, tables, charts, images, graphs, chemical molecular structural formulas, mathematical formulas, etc.) from various types of documents into data by utilizing Deep Document Understanding (DDU) technology.

[0116] At this time, the deep document understanding technology according to the present invention can identify a plurality of layout areas constituting a document by using a document analysis model. Such a document analysis model may also be referred to as a “Document Layout Analysis (DLA) model.” Furthermore, the deep document understanding technology can structurally analyze multimodal information contained in a document based on the content contained in each identified layout area. Specifically, the document analysis model according to the present invention may refer to an artificial intelligence model trained to output layout area data for said document from at least one input document.

[0117] After a document analysis model is trained using a training document dataset as described above, the present invention may perform document understanding processing on an actual document to be analyzed using the trained document analysis model. Below, a process is described regarding an inference step distinct from the training process, in which the document analysis model analyzes a document to identify a layout area, extracts content included in the corresponding layout area, and generates output data based on the extracted content.

[0118] Specifically, the document analysis model may be configured to perform the role of identifying (or detecting) at least one and / or at least one layout area containing at least one and / or at least one piece of content through analysis of the layout (or structure) of the document. In this case, the present invention may generate augmented document data used for training the document analysis model. Here, the augmented document data may refer to document data generated such that it includes the same layout area as the training document data, but the content included in the layout area contains alternative content different from that of the training document data.

[0119] Meanwhile, in this specification, the term “document analysis model” may refer to a processing module that receives a document as input and analyzes the structural arrangement, layout configuration, or content distribution of the document. The document analysis model may include a rule-based processing module, a statistical processing module, a machine learning-based model, a neural network model, or a combination thereof, and may be composed of a single model or operate by combining multiple models. The document analysis model may be a pre-trained model or may be a model that operates according to rules or settings in an untrained state.

[0120] The present invention proposes a document understanding method and system capable of providing output data for at least one document by structurally analyzing multimodal information contained in at least one document using a document analysis model trained on a training document dataset including augmented document data.

[0121] Meanwhile, the document understanding system according to the present invention may include at least one artificial intelligence model. In this case, the at least one artificial intelligence model mentioned in the present invention may include various models that can be utilized depending on various situations or purposes. For example, the artificial intelligence model of the present invention may include at least one of a Vision Language Model (VLM), a Machine Learning (ML) model, a Deep Learning model, a Deep Neural Network (DNN), a Language Model (LM), a Large Language Model (LLM), a Super-large Foundation model, a Generative AI model, a Transformer-based model, a Supervised Learning (SL) model, a Reinforcement Learning (RL) model, and a special purpose model (e.g., a time series forecasting model (e.g., ARIMA model, SARIMA model, etc.), a time series Foundation model, a Graph Neural Network, a Multimodal Model, a Natural Language Processing (NLP) model, a computer vision model, a speech recognition / synthesis model, a recommendation system model, etc.).

[0122] Furthermore, the artificial intelligence model used in the document understanding system according to the present invention may be implemented with a single or multiple components. More specifically, the artificial intelligence model used in the present invention may be implemented with a single component (1) or with multiple components (2, 3, 4 or more, etc.).

[0123] In this regard, when multiple artificial intelligence models are implemented in the present invention, the multiple artificial intelligence models may be implemented as models of the same type. In this case, the multiple artificial intelligence models may be implemented to perform different functions (or roles) in the present invention.

[0124] In the foregoing, the document understanding method according to the present invention has been generally described, and this can be implemented by the document understanding system described below.

[0125] Hereinafter, the document understanding system according to the present invention will be examined in more detail together with the attached FIGS. 4, 5a, and 5b. FIGS. 4, 5a, and 5b are conceptual diagrams for explaining the document understanding system according to the present invention.

[0126] Meanwhile, as illustrated in FIG. 5, the document understanding system (100) according to the present invention may include at least one of a communication unit (110), a storage unit (120), a control unit (130), and a document understanding unit (140). However, the components of the document understanding system (100) according to the present invention are not limited thereto and may further include various hardware components that perform the same or similar roles as described in the description of the present specification.

[0127] Although not illustrated, the document understanding system (100) according to the present invention may include at least one processor, and such processor may include one or more general-purpose processors and / or one or more special-purpose processors (e.g., digital signal processor, tensor processing unit (TPU), graphics processing unit (GPU), neural network processing unit (NPU), application integrated circuit, application semiconductor (ASIC), field programmable gate array (FPGA), quantum processing unit (or quantum processor, QPU), etc.). At least one processor may be configured to execute instructions, computer-readable instructions, and / or other instructions described herein that are stored (or included) in the storage unit (120). The document understanding method and system according to the present invention may perform data processing described below in cooperation with memory and at least one processor. The processor may perform a series of operations and data processing using data and information stored in memory. In this case, memory may be a component of the storage unit (120).

[0128] In addition, the document understanding system (100) according to the present invention can perform data processing and computation processes using quantum gates, quantum entanglement, and quantum superposition states, taking into consideration implementation in a quantum computer environment. For example, the present invention can perform parallel computations based on qubits, and such quantum computations can operate complementarily with existing classical computers.

[0129] Such quantum computers may include parallel computation using qubits and high-speed data processing devices utilizing quantum entanglement, and hardware-based computational optimization using FPGAs and ASICs is possible. In addition, quantum computers may utilize quantum processors capable of qubit-based parallel computation, and data processing efficiency can be improved through a hybrid structure with existing classical computers.

[0130] Meanwhile, the document understanding system (100) according to the present invention may exist inside a server (hereinafter referred to as the server) established to perform a specific purpose (e.g., deep document understanding, providing output data for at least one document, and generating augmented document data), or it may exist as a separate system from said server. When the document understanding system (100) exists inside the server, it may provide various services related to the present invention (e.g., providing output data for at least one document and generating augmented document data, etc.) through at least one component located inside the server, or through configuration modules that perform functions similar to said components. In this case, the application may provide various services related to the present invention on an electronic device on which the application is installed through communication with the server.

[0131] Meanwhile, the communication unit (110) according to the present invention may be connected to a user terminal (10), an external server, a central server, a device, and at least one network via a wireless or wired network, and may be configured to receive or transmit overall data and information necessary for the operation of the document understanding system (100) according to the present invention.

[0132] The communication unit (110) can receive a user query from the user terminal (10). Here, “receiving a user query” may mean receiving an input signal corresponding to the user query based on the user query being input by the user through the input unit configuration provided in the user terminal (10). Here, the user terminal (10) may include at least one of a mobile phone, a smartphone, a notebook computer, a laptop computer, a slate PC, a tablet PC, an ultrabook, a desktop computer, a digital broadcasting terminal, a PDA (personal digital assistants), a PMP (portable multimedia player), a navigation device, a wearable device (e.g., a smartwatch, a smart glass, a head-mounted display).

[0133] In addition, the input section of the user terminal (10) in the present invention does not necessarily refer to a hardware means, but can be understood as a channel for receiving input from a user. Such an input section of the user terminal (10) may also be referred to as a user interface module. The input section of the user terminal (10) may include a touch screen, a computer mouse, a keyboard, a keypad, a touchpad, a trackball, a joystick, a voice recognition module, or other similar devices. However, the present invention does not limit the type of input section of the user terminal (10). Here, user input may include documents, text, images (or videos), voice, etc. In this case, the document understanding system (100) may further include a module that converts voice into text.

[0134] Meanwhile, the communication unit (110) may include at least one communication module capable of wireless communication and wired communication between the document understanding system (100) and at least one communication target. Additionally, the communication unit (110) may include a communication module that connects the document understanding system (100) to at least one network.

[0135] Meanwhile, the communication unit (110) can support various communication methods depending on the communication standard of the device being communicated. For example, the communication unit (110) can be configured to communicate with a communication target using at least one of the following technologies: WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), 5G (5th Generation Mobile Telecommunication), Bluetooth (Bluetooth™ RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra-Wideband), ZigBee, NFC (Near Field Communication), and Wireless USB (Wireless Universal Serial Bus).

[0136] Next, the document understanding system (100) according to the present invention can output information through an output unit configuration (e.g., a display unit, a touch screen, a speaker, etc.) provided in a user terminal (10) linked to the document understanding system (100). For example, the document understanding system (100) can output at least one page (2000, or a service page) linked to the document understanding system (100) according to the present invention to the display unit of the user terminal (10).

[0137] Next, the storage unit (120, or memory) serves to store various data related to the present invention, and the storage unit (120) may be provided in the document understanding system (100) itself, or alternatively, at least part of the storage unit (120) may mean a database (DB) or memory. The storage unit (120) may include one or more non-transient computer-readable storage media that can be read and / or accessed by at least one processor. One or more computer-readable storage media may include volatile and / or non-volatile storage components such as optical, magnetic, organic, or other memory or disk storage devices. In some examples, the storage unit (120) may be implemented using a single physical device (e.g., one optical, magnetic, organic, or other memory or disk storage device), whereas in other examples, the storage unit (120) may be implemented using multiple physical devices.

[0138] The storage unit (120) may include computer-readable instructions and additional data. The storage unit (120) may include a storage necessary to perform at least some of the methods, scenarios, and techniques described in this specification and / or at least some of the functions of the device and network. Furthermore, at least some of the storage unit (120) may be a cloud storage or a cloud server. That is, the storage unit (120) is sufficient as a space where information necessary for the operation of the document understanding system (100) according to the present invention is stored, and it can be understood that there are no restrictions on the physical space. Accordingly, the storage unit (120) and the database may be used interchangeably without being separately distinguished below.

[0139] The storage unit (120) may store at least one document collected (or received) from various sources (e.g., a web corpus, a document corpus, a database (DB) website, an API, a server linked to the document understanding system (100), a central server, an external server, cloud storage, a user terminal (10), a large dataset, etc.). For example, the storage unit (120) may store at least one and / or at least one or more (multiple) documents collected from at least one of the various sources (e.g., a user terminal (10)). In this case, the collected documents may include documents related to at least one domain (or field). Alternatively, the collected documents may include documents containing at least one content.

[0140] Additionally, the storage unit (120) may store a training document data set for training the document analysis model (141). Specifically, the storage unit (120) may store a training document data set including at least one augmented document data.

[0141] That is, the storage unit (120) is sufficient as a space where information necessary for the operation of the document understanding system (100) according to the present invention is stored, and it can be understood that there are no restrictions on the physical space.

[0142] Furthermore, the storage unit (120) may store a computer program including computer program instructions. The storage unit (120) may store a computer program including computer program instructions that control the operation of the system (100) or control the operation of the control unit (130) when loaded into the processor of the document understanding system (100). Specifically, instructions for the operation of the document data generation module (131) and the document understanding unit (140) may be stored in the storage unit (120).

[0143] Next, the document understanding unit (140) may be configured to effectively understand complex forms of information contained in various types of documents. The document understanding unit (140) may be configured to recognize the structure of at least one document (20) and to recognize the relationships of content (e.g., Image-Text, Image-Image, Text-Text) contained in at least one document (20) to perform the role of extracting information. In the present invention, the document understanding unit (140) may also be named a “deep document understanding model,” a “document understanding model,” or a “DDU model.”

[0144] The document comprehension unit (140) can be configured to extract various forms of content (e.g., layout, text, table, chart, image, graph, chemical molecular structural formula, mathematical formula, etc.) from at least one and / or one or more documents (e.g., paper, book, patent document, report, etc.).

[0145] More specifically, the document comprehension unit (140) may be a model trained to understand structured data, unstructured data, linguistic data (or linguistic elements) and non-linguistic data (or non-linguistic elements), etc., included in the document (20), and to extract various content and / or knowledge based on the understood content.

[0146] In one embodiment, the document understanding unit (140) can understand the chemical structure of a molecular structural formula included in at least one document (20) and, based on the understood result, convert the molecular structural formula into a SMILES string expression and extract it. Additionally, the document understanding unit (140) can understand the chemical structure of a molecular structural formula and, based on the understood result, perform a graph conversion corresponding to the molecular structural formula.

[0147] In another embodiment, the document understanding unit (140) can understand texts associated with a molecular structural formula among texts included in at least one document (20) and extract them as text data associated with the molecular structural formula.

[0148] In another embodiment, the document comprehension unit (140) can recognize rows and columns constituting a table associated with a molecular structural formula from at least one document (20) and convert them into structured data in a format such as HTML or Excel to extract them.

[0149] Additionally, the document understanding unit (140) can extract relationship information (or relationships) between molecular structures included in at least one document (20).

[0150] In one embodiment, the document understanding unit (140) can understand the relationship between a first molecular structure and a second molecular structure included in at least one document (20) and extract relationship information in which a third molecular structure is generated through a chemical reaction between the first molecular structure and the second molecular structure. In this case, the relationship information between the molecular structures can be extracted by understanding the text included in at least one document (20) to be analyzed, or by understanding non-verbal data included in said at least one document (20).

[0151] In another embodiment, the document understanding unit (140) can understand the relationship between the first molecular structure and the second molecular structure through a symbol (e.g., plus sign, arrow, etc.) located in one region among a plurality of regions included in at least one document (20) where the first molecular structure and the second molecular structure are located, and can extract relationship information in which a third molecular structure is generated through a chemical reaction between the first molecular structure and the second molecular structure.

[0152] Furthermore, the document comprehension unit (140) can extract various forms of content satisfying established content criteria from at least one and / or at least one document. Here, the established content criteria can be set in various ways and can be determined according to the purpose or use of the document comprehension system (100). For example, if the purpose of use of the document comprehension system (100) is chemistry, bio, new materials, new substances, and new drug development, the document comprehension unit (140) can be trained to understand and extract content related to chemistry, bio, new materials, new substances, and new drug development from at least one document (20). In this case, the established content criteria may include content related to molecular structures related to at least one of chemistry, bio, new materials, new substances, and new drug development. Here, the document comprehension unit (140) can extract content related to chemistry, bio, new materials, new substances, and new drug development from at least one document (20) according to the established content criteria. However, this is merely one embodiment, and the established content standards in the present invention are not necessarily limited thereto.

[0153] As described above, the document comprehension unit (140) can convert various forms of content (or information, data, etc.) contained in at least one document (20) into data that can be understood by a machine and / or artificial intelligence model. The data extracted using the document comprehension unit (140) can be sorted by page or document and stored in a storage unit (120, or memory). In this case, the document comprehension unit (140) can be configured (or constructed) to include at least one and / or at least one or more (multiple) artificial intelligence models (or models, modules, etc.).

[0154] Here, artificial intelligence models may include various models that can be utilized depending on various situations or purposes. For example, an artificial intelligence model may include at least one of a machine learning (ML) model, a deep learning model, a deep neural network (DNN), a language model (LM), a large language model (LLM), a massive foundation model, a generative AI model, a transformer-based model, a supervised learning (SL) model, a reinforcement learning (RL) model, a vision-language model (VLM), and a special-purpose model (e.g., a time series forecasting model (e.g., ARIMA model, SARIMA model, etc.), a time series foundation model, a graph neural network (GNN), a multimodal model, a natural language processing (NLP) model, a computer vision model, a speech recognition / synthesis model, a recommendation system model, etc.).

[0155] In this regard, as illustrated in FIG. 5a, the document understanding unit (140) may include at least one of a document analysis model (e.g., “DLA”, 141), a text extraction model (e.g., “OCR”, 142), a table extraction model (e.g., “Table”, 143), a chart extraction model (e.g., “Chart”, 144), a molecular structure formula extraction model (e.g., “Mol detect”, 145), a chemical reaction information extraction model (e.g., “Reaction”, 146), a molecular structure formula conversion model (e.g., “Chemical Structure Processing”, 147), a content order sorting model (e.g., “ROD”, 148), and a multimodal model (e.g., “VLM”, 149). However, at least some of the plurality of models (141 to 149) may be implemented as separate components from the document understanding unit (140). Alternatively, at least some of the plurality of models (510, 520, 530, 540, 550, 560, 570, 580, 590) may be implemented as part of the document understanding system (100). The present invention is not limited to any one of these. In addition, the term “model” in the present invention may also be referred to as “module.”

[0156] In the present invention, the document understanding system (100) can identify (or identify) the extension (e.g., PDF, PPTX, HWPX, PNG, JPG, etc.) of at least one document (20) to be analyzed based on the identification of at least one document (20), and convert the document (or each page of the document) into a pre-set specific format. For example, the pre-set specific format (or format, shape, etc.) may include an image.

[0157] Additionally, the document understanding system (100) can process a document converted into a specific format (i.e., a document converted into an image) as input to a document analysis model (141) included in the document understanding unit (140). The document analysis model (141) can be configured to perform the role of analyzing the structure of the document and separating the background.

[0158] Specifically, the document analysis model (141) may be configured to perform the role of identifying (or detecting) at least one and / or at least one layout area containing at least one and / or at least one content through analysis of the structure (or layout) of the document. Here, the identification of the layout area (or layout identification) may be a process of finding the location of an area (or object) corresponding (or for) each of a plurality of contents having different data characteristics (e.g., text, table (or chart), chart (or graph), molecular structural formula, chemical reaction information, etc.).

[0159] In this specification, the term “layout area” may refer to a spatial or logical unit in which content is arranged within a document. The layout area may be classified according to at least one criterion among location, size, shape, inclusion relationship, adjacency relationship, or reading order on the document, and may be a physically continuous area or a logically related area. A single document may contain multiple layout areas having different attributes.

[0160] In this specification, “layout area data” may refer to information for representing layout areas identified from a document in the form of data. The layout area data may include location information, size information, category information, score information, or structural relationship information corresponding to each layout area, and may be configured to include all or part of such information. The layout area data may be used as reference data in subsequent content extraction or processing steps.

[0161] In this specification, “content” may refer to a semantic unit included within a document and may include text, tables, images, figures, formulas, charts, molecular structural formulas, or chemical reaction information. Additionally, “content data” may refer to information in the form of data generated as a result of extracting or processing the content. Content data may include part or all of the original content and may be in a form that has undergone preprocessing or postprocessing.

[0162] In this specification, the term “data characteristics” may refer to structural, formal, or semantic attributes of content or content data. The data characteristics may include at least one of the type of content, format of expression, resolution, language, symbol composition, semantic category, or processing difficulty, and may be defined by a combination of multiple criteria rather than being limited to a single criterion.

[0163] More specifically, the document analysis model (141) analyzes the document to include i) text (e.g., Body-text (e.g., a general body paragraph), Caption (e.g., special text located outside of a figure, table, chart, etc., and describing the element), List-item (e.g., an element of a list (or catalog) hanging, i.e., the paragraph is indented more from the second line onwards than the first line. In this case, each bbox has an entity, and each entity has a category), Title (e.g., the overall title of the document, generally displayed in a large font on the first page), Section-header (e.g., representing all types of text titles excluding the overall document title, in this case, each bbox has an entity, and each entity has a category), Footnote (e.g., small text generally located at the bottom of the page, containing numbers or symbols mentioned in the text above), Formula (e.g., a mathematical equation existing on its own line), Page-header (e.g., a repeating element such as a page number displayed at the top outside the general text flow), Layout areas including at least one of the following can be identified: ii) page-footer (e.g., repeating elements such as page numbers outside the general text flow), iii) figures (e.g., graphics, photos, etc.), iv) charts (e.g., xy graphs, bar charts, line charts, etc.), iv) tables (e.g., data arranged in a row and column structure), v) other visual content (e.g., flowcharts, diagrams, infographics, maps, organizational charts, etc.).

[0164] At this time, when the document analysis model (141) identifies a layout area containing text, if the text is included in the continuous content flow (body flow) of the document, it may identify it as a layout area containing normal text. On the other hand, if the text is a page component that appears repeatedly outside the flow of the document, the document analysis model (141) may identify it as a layout area containing format text. For example, normal text is text that constitutes the actual content of the document and may include at least one of body text, caption, list item, title, section header, footnote, and formula (or mathematical expression). In contrast, format text is text for page formatting rather than the content of the document and may include at least one of page header and page footer.

[0165] A document analysis model (141) can analyze a document converted into a specific format (or a document of a specific format) to identify a plurality of layout areas each containing a plurality of contents (or at least one content) having different data characteristics. In this case, the plurality of layout areas may include at least one of a layout area containing a caption (e.g., “Caption”), a layout area containing a page header (e.g., “Page-header”), a layout area containing a table (e.g., “Tabel”), a layout area containing body text (e.g., “Body-text”), a layout area containing a list item (e.g., “List-item”) (615), and a layout area containing a chart (e.g., “Chart”). For example, “idx” included in the document layout analysis result (e.g., “Document Layout Analysis”) may represent a unique number used when referencing the corresponding layout area, and “bbox” may represent location information that each layout area (or object) occupies within the page of the document. For the sake of convenience of explanation, the present invention is described as identifying multiple layout areas, each containing multiple contents; however, the present invention can identify a single layout area containing a single content by analyzing a document. Accordingly, 'multiple contents' and 'at least one content' according to the present invention may be used interchangeably.

[0166] That is, the document analysis model (141) can receive an image of each page of a document as input and output at least one of a label corresponding to each of the multiple contents (e.g., text label, table label, chart label, etc.), location information of each of the multiple contents (e.g., bounding box coordinates in the format [x1, y1, x2, y2]), and a score for each of the multiple contents (e.g., a probability value indicating whether the identified content belongs to the content). The output of this document analysis model (141) can also be understood as a “document layout identification result” or a “document layout analysis result.”

[0167] Subsequently, the document comprehension unit (140) can extract multiple contents included in each of the multiple layout areas by utilizing multiple models specialized for each of the multiple contents. To this end, the document comprehension unit (140) can process multiple layout areas containing multiple contents as inputs for each of the multiple models. In the present invention, processing multiple layout areas as inputs for each of the multiple models can also be interpreted to mean calling models to extract multiple contents based on the output of the document analysis model (141). Alternatively, it can be interpreted to mean processing at least one of the output of the document analysis model (141) and a document of a specific format as inputs for each of the multiple models.

[0168] The document comprehension unit (140) can process at least one of a document of a specific format, including the output of a document analysis model (141) (e.g., labels corresponding to each of a plurality of contents, location information of each of a plurality of contents within a document (or image), etc.) and a plurality of layout areas, as an input to each of a plurality of models.

[0169] In this specification, the term “multiple models” may mean two or more processing models configured to perform the same or different functions. The multiple models may be selectively applied depending on the data characteristics of the content, may operate in parallel or sequentially, and the outputs of each model may be combined to produce a single processing result.

[0170] First, when a document of a specific format is input, the text extraction model (142) can recognize a layout area containing general text and a layout area containing format text from a document of a specific format, respectively, based on the output of the document analysis model (141) (e.g., a label corresponding to general text and location information of general text within the document of a specific format, a label corresponding to format text and location information of format text within the document of a specific format, etc.). Then, the text extraction model (142) can extract text from each of the multiple layout areas containing text. This text extraction model (142) can also be understood as a model that performs Optical Character Recognition (OCR), which converts characters (or text images) contained in an image or document into text data that can be recognized and read by a machine (or computer). In the present invention, the text extraction model (142) may also be named an “OCR model,” a “text model,” a “text processing model,” a “text understanding model,” or a “text specialized model,” etc.

[0171] Additionally, when a document of a specific format is input, the table extraction model (143) can recognize a layout area containing a table from a document of a specific format based on the output of the document analysis model (141) (e.g., a label corresponding to the table and location information of the table within the document of the specific format). Then, the table extraction model (143) can extract a table from the layout area containing the table. In the present invention, the table extraction model (143) may also be named a “table model,” a “table processing model,” a “table understanding model,” or a “table specialized model.”

[0172] Additionally, the table extraction model (143) may be configured to perform table structure recognition (TSR). Table structure recognition may mean extracting the logical and physical structure of an unstructured table image into a machine-readable format. For example, the table extraction model (143) may recognize the logical and physical structure of a table and convert the table into a pre-set specific format (e.g., “HTML”).

[0173] Furthermore, when a document of a specific format is input, the chart extraction model (144) can recognize a layout area (617) containing a chart from a document of a specific format based on the output of the document analysis model (141) (e.g., a label corresponding to the chart and location information of the chart within the document of the specific format). Then, the chart extraction model (144) can extract a chart from the layout area containing the chart. In the present invention, the chart extraction model (144) may also be named a “chart model,” a “chart processing model,” a “chart understanding model,” or a “chart specialized model.”

[0174] Additionally, the chart extraction model (144) can perform the role of analyzing the chart to extract raw data of the chart. For example, the chart extraction model (144) can process the chart to structure information about the entire chart into a specific pre-set format (e.g., “JSON”) and generate (or output) structured data based on this. Additionally, the chart extraction model (144) can reconstruct the visual results of the chart based on the structured data to generate (or output) reconstructed data. Additionally, the chart extraction model (144) can recognize numerical information such as axis labels and data points from a layout area containing the chart and convert it into numerical data in the form of a table.

[0175] Meanwhile, if a document received from a user terminal (10) contains content (or data, information, etc.) related to a chemical domain (or chemical field), the document understanding system (100) may input the document into at least one model specialized in the chemical domain. For example, in the present invention, the model specialized in the chemical domain may include at least one of a molecular structure formula extraction model (145), a chemical reaction information extraction model (146), and a molecular structure formula conversion model (147).

[0176] First, the document understanding system (100) can process a document of a specific format as input to a molecular structure formula extraction model (145) in order to extract at least one molecular structure formula among a plurality of contents included in the document.

[0177] Here, the molecular structure extraction model (145) may be configured to accurately detect regions corresponding to (or corresponding to) molecular structure formulas within a document. More specifically, the molecular structure extraction model (145) may perform molecular detection (or molecular detection) by identifying and locating at least one and / or at least one molecular structure formula contained in a document. This molecular structure extraction model (145) may be a model trained using at least one training document dataset to accurately detect molecular structure formulas.

[0178] In one embodiment, when a document of a specific format is input, the molecular structure formula extraction model (145) recognizes a layout area containing a molecular structure formula and can extract a molecular structure formula from the recognized layout area. In the present invention, the molecular structure formula extraction model (145) may also be named a “molecular structure formula specialized model,” a “molecular structure formula model,” a “molecular structure formula processing model,” or a “molecular structure formula understanding model,” etc.

[0179] Additionally, the document understanding system (100) can process a document of a specific format as input to a chemical reaction information extraction model (146) in order to extract at least one chemical reaction information among a plurality of contents included in the document.

[0180] Here, the chemical reaction information extraction model (146) may be configured to perform the role of extracting chemical reaction information consisting of reactants, reaction conditions, and products within a document. The chemical reaction information extraction model (146) may perform the role of identifying the location of these components from each chemical reaction information and assigning an appropriate class to the corresponding area. For example, the chemical reaction information extraction model (146) may perform reaction parsing (or reaction parsing) to extract and classify at least one and / or at least one chemical reaction information contained in a document. This chemical reaction information extraction model (146) may be a model trained using at least one training document dataset to extract and classify reaction roles within a chemical reaction diagram.

[0181] In one embodiment, when a document of a specific format is input, the chemical reaction information extraction model (146) recognizes a layout area (not shown) containing chemical reaction information and can extract chemical reaction information from the recognized layout area. In the present invention, the chemical reaction information extraction model (146) may also be named a “chemical reaction information specialized model,” a “chemical reaction information model,” a “chemical reaction information processing model,” or a “chemical reaction information understanding model,” etc.

[0182] However, in the present invention, recognizing a layout area containing molecular structural formulas and chemical reaction information included in a document, and extracting molecular structural formulas and chemical reaction information from the recognized layout area, can be performed by a document analysis model (141).

[0183] Furthermore, the molecular structure formula conversion model (147) may be configured to perform the role of converting the molecular structure formula into at least one pre-set chemical structure representation format (e.g., SMILES, InChI, Mol, etc.). For example, after recognizing the molecular structure formula, the molecular structure formula conversion model (147) may analyze the bonding relationships between the atoms constituting the molecular structure formula and said atoms, and convert the molecular structure formula into at least one chemical structure representation format. Here, the chemical structure representation format may mean that the molecular structure formula has been converted into a representation that a machine (or computer) can understand and use.

[0184] In this case, the document understanding unit (140) inputs the molecular structural formula extracted from the molecular structural formula extraction model (145) into the molecular structural formula conversion model (147) and can obtain a chemical structure representation format for the molecular structural formula converted through the molecular structural formula conversion model (147). Alternatively, the document understanding system (100) inputs a document of a specific format into the molecular structural formula conversion model (147) and can recognize the molecular structural formula contained in the document of the specific format in the molecular structural formula conversion model (147) and convert it into a chemical structure format.

[0185] Next, the content order sorting model (148) may be configured to perform the role of specifying (or determining) order information of at least one and / or at least one content extracted from the document.

[0186] Additionally, the content order sorting model (148) may be configured to perform the role of sorting at least one and / or at least one piece of content extracted from a document in a pre-set order (e.g., human reading order). This content order sorting model (148) may also be understood as a model that performs Reading Order Detection (ROD). Here, Reading Order Detection may mean sorting (or arranging) text in a document (or document image) having various layouts in a logical order in which a human reads. In one embodiment, Reading Order Detection may mean a rule for sorting multiple text areas (or blocks) within a document in the order in which a human actually reads. In the present invention, the expression "sorting content in order" may also be referred to as "arranging content in order" or "arranging content in order."

[0187] As seen above, the document analysis model (141) identifies multiple layout areas each containing multiple contents through analysis of each page of the document. At this time, in order to understand a document that is visually diverse, it is necessary to organize the contents included in the multiple layout areas in the same way as a human reads.

[0188] To this end, the content order sorting model (148) can perform a task of arranging the content extracted from the document according to a pre-set order. Specifically, the content order sorting model (148) can arrange the order of the content in the order of reading, starting from the first layout area located at the top-left of the document page. At this time, since the content order sorting model (148) arranges (or reconstructs) the order according to the flow of the document body (or body text flow), i) figures, tables, charts, ii) format text such as page headers, page footers, journal names, etc., may be excluded during the content order sorting process. Here, the text included in the fifth layout area and the text included in the tenth layout area may have a structure that follows the same paragraph or a continuous sentence. In this case, the content order sorting model (148) can arrange the text included in the tenth layout area so that it follows the text included in the fifth layout area.

[0189] That is, the content ordering model (148) can arrange multiple layout areas (e.g., layout areas containing text) within a page of a document in a pre-set order (e.g., left → right, top → bottom, etc.), and when content is consecutive, such as with the 5th layout area and the 10th layout area, the two layout areas can be arranged to be connected continuously. However, in the present invention, the arrangement of the content order can also be performed by a multimodal model (149).

[0190] Next, the multimodal model (or multimodal artificial intelligence model, 590) may include a Vision-Language Model (VLM) that simultaneously understands and processes text and images. This multimodal model can integrally understand visual information (or visual content, visual elements) and text information (or text content, text elements, linguistic information, linguistic elements, etc.) contained in a document, and perform data processing to provide an optimal output (or output data, output result, outcome) according to a user request (or user purpose). In the present invention, the multimodal model (149) may also be named a “vision language model,” a “generative artificial intelligence model,” or a “generative model.”

[0191] In one embodiment, the multimodal model (149) can simultaneously understand text content and visual content (e.g., pictures, tables, charts, molecular structural formulas, etc.) included in each page (or image data) of the input document and / or converted document, identify semantic relationships between the contents, and perform data processing to provide optimal output according to user requests. For example, the data processing may include at least one of i) converting the document into an editable form, ii) converting the document into a form that meets the user's request (or needs), iii) summarizing the document, iv) translating the document, v) structuring the document into semantic units to generate structured data, or vi) reconstructing the document to generate a reconstructed document.

[0192] In this case, the input data of the multimodal model (149) may be data of a specific format (or form, style). For example, the specific format may include JSON (JavaScript Object Notation). This specific format of data may refer to a data format for structuring and expressing a layout area containing each of multiple contents extracted from a document, and order information, type information, characteristics (or attribute information), location information, etc., for each of the multiple contents. This specific format of data may be generated based on the output of the document analysis model (141) (e.g., identification result of multiple layout areas) and the output of the content order sorting model (148) (e.g., order of contents). Through this, the relationships between contents and contextual flow within the document are organized into a form that can be mechanically interpreted, and can be utilized as input data to enable the multimodal model (149) to perform processes such as converting the document into an editable form, reconstructing the document, converting the document into a form that meets the user's request, or translating the document in subsequent processes. Additionally, data of a specific format can be utilized as information necessary to rearrange or visualize content within a document in a user interface. This data of a specific format can be understood as data generated by a document understanding system (100) or a document understanding unit (140).

[0193] Meanwhile, in the present invention, depending on the purpose or use of the document understanding system (100), it may be implemented with at least one of the plurality of models (141 to 149) described above excluded. Alternatively, in the present invention, depending on the purpose or use of the document understanding system (100), the user may disable (i.e., turn OFF) at least one of the plurality of models (141 to 149) so that it is not calculated and / or called during the document processing process. That is, in the present invention, by allowing a model to be selected according to the purpose or use of the document understanding system (100), the processing speed and / or resource usage of the system can be optimized.

[0194] Additionally, the present invention may be implemented to include at least one additional model depending on the purpose or use of the document understanding system (100). Alternatively, the present invention may provide an environment in which a user can additionally configure a model depending on the purpose or use of the document understanding system (100).

[0195] In one embodiment, the document comprehension unit (140) may further include a style module (e.g., “style”, 575) capable of processing a document by reflecting user's preferred style information (e.g., text style (font (or typeface)), character size, line spacing, boldness, etc.), writing style and / or tone style (e.g., tone of voice, sentence length, summary level, etc.), format style (e.g., table style, PPT style, report / paper, patent document style, etc.), and color and / or design (e.g., color tone, accent color type, background brightness, etc.).

[0196] Next, the control unit (130) can perform the role of controlling the overall operation of the document understanding system (100) related to the present invention. The control unit (130) can process signals, data, information, etc. that are input or output through the components of the document understanding system (100) described above, or perform a series of data processing to provide or process appropriate information and functions to the user. The control unit (130) can be physically implemented by the processor described above.

[0197] Meanwhile, the control unit (130) may include a document data generation module (131) capable of generating augmented document data. In this case, although the present invention describes the control unit (130) as including the document data generation module (131), it is not limited thereto. If the document data generation module (131) exists on an external server, the control unit (130) may call or control the functions of the document data generation module (131) by interacting with the external server.

[0198] As illustrated in FIG. 5b, the control unit (130) can generate at least one augmented document data using at least one training document data (500). Specifically, at least one training document data (21) may be composed of at least one layout area (510, 520, 530). At this time, each of the at least one layout area may include at least one specific content (511, 521, 531). The control unit (130) can disable the specific content included in the at least one layout area constituting the at least one training document data (500) using a document data generation module (131). Here, disabling the content may mean controlling so that the content of the specific content is not displayed in the at least one layout area by at least one of masking, blurring, hiding, alpha adjustment, and color overwrite.

[0199] The method for disabling specific content included in at least one layout in the present invention is not limited to the examples described above, and may further include various methods to prevent the content of specific content from being displayed in at least one layout.

[0200] Furthermore, the control unit (130) can generate augmented document data (22) including alternative content (512, 522, 532) having the same data characteristics as specific content in at least one deactivated layout area (510, 520, 530). Such augmented document data (22) can be used as training data for a document analysis model (141) according to the present invention.

[0201] As described above, the present invention provides a document understanding system (100) capable of effectively understanding various types of documents. More specifically, the document understanding system (100) according to the present invention can provide a service and / or function that automatically converts multimodal information from various types of documents into data using Deep Document Understanding (DDU) technology. Below, we will examine the document understanding method and document analysis model in more detail with reference to the attached FIGS. 6, 7, 8a, and 8b. FIGS. 6 is a flowchart for explaining the document understanding method according to the present invention, and FIGS. 7, 8a, and 8b are flowcharts for explaining the document understanding method according to the present invention.

[0202] Meanwhile, in the present invention, a process of specifying at least one document to be analyzed may be performed (S610, see FIG. 6).

[0203] The control unit (130) can collect at least one document to be analyzed among a plurality of documents collected (or received) from various sources (e.g., a web corpus, a document corpus, a database (DB) website, an API, a server linked to the document understanding system (100), a central server, an external server, cloud storage, a user terminal, a large dataset, etc.).

[0204] Specifically, the control unit (130) can identify at least one document to be analyzed in response to receiving at least one document from the user terminal (10).

[0205] For example, the control unit (130) may provide at least one page (2000) to the user terminal (10) for receiving at least one document to be analyzed. Based on (or derived from) user input received from the user terminal (10), the control unit (130) may identify the received at least one document as at least one document to be analyzed.

[0206] As another example, the control unit (130) may collect multiple documents from an external source server and identify at least one document among the multiple documents to be analyzed. As an example, the control unit (130) may collect multiple documents from one of a website server, a portal site server, a blog platform server, a cafe platform server, a community server, a news article provider server, a social media (SNS) server, a video platform server, a shopping mall server, a review platform server, an in-house document management server (DMS), a cloud storage server, a database (DB) server, and an Open API server. Specifically, the control unit (130) may collect multiple documents using at least one method among web crawling and web scraping, and identify at least one document among the at least multiple documents to be analyzed.

[0207] At this time, the method for specifying the document to be analyzed in the present invention is not limited to the examples described above, and can be implemented in a wide variety of ways so that various modifications are possible depending on at least one of the type of document to be analyzed, the collection path, and the user query.

[0208] That is, in the present invention, the control unit (130) may specify at least one document to be analyzed based on a document directly input from a user terminal, may specify at least one document to be analyzed based on a plurality of documents collected from an external source server, or may specify at least one document retrieved from a previously stored database. Furthermore, it can be understood that the control unit (130) may specify a document to be analyzed consisting of at least one sentence by using all methods, whether using these methods alone or in combination with each other.

[0209] As illustrated in FIG. 7, the control unit (130) can identify at least one document (or document to be analyzed, 20) among a plurality of documents collected from a plurality of sources. For example, the control unit (130) can identify at least one document (20) to be analyzed from at least one of a user terminal (10), at least one web page (701), at least one link content (702), and at least one application (703).

[0210] For example, the control unit (130) may provide at least one page (2000) to the user terminal (10), which includes an input area for receiving at least one document (20) to be analyzed. The control unit (130) may receive at least one document to be analyzed based on (or derived from) user input received from the user terminal (10). User input according to the present invention may include at least one of multimodal input data consisting of text, images, videos, voice, and combinations thereof, including the document to be analyzed. Specifically, the control unit (130) may identify at least one document (20) based on user input received from the user terminal (10).

[0211] Next, in the present invention, a process of inputting at least one document into a document analysis model trained based on a training document dataset containing at least one augmented document data may be performed (S620, see FIG. 6). Furthermore, in the present invention, a process of analyzing at least one document in the document analysis model and identifying at least one layout area having different attributes from at least one document may be performed (S630, see FIG. 6).

[0212] As illustrated in FIG. 8a, the document analysis model (141) according to the present invention may be configured to perform the role of identifying (or detecting) at least one and / or at least one layout area (811 to 817, 820, 830) containing at least one and / or at least one content through analysis of the structure (or layout) of the document. Here, the layout area identification (or layout identification) may be a process of finding the location of an area (or object) corresponding to (or about) each of a plurality of contents having different data characteristics (e.g., text, chart (or graph), image, molecular structural formula, chemical reaction information, etc.).

[0213] Specifically, the document comprehension unit (140) can identify the extension (e.g., PDF, PPTX, HWPX, PNG, JPG, etc.) of at least one document (20) that has been identified, and convert the at least one document (20) with the identified extension into a specific format (or form, style, etc.) that has been set. For example, the specific format that has been set may include images, and this may be a process of converting at least one and / or at least one page included in at least one document into an image.

[0214] The above document analysis model (141) can generate layout area data corresponding to each layout area when analyzing a document to identify multiple layout areas. The layout area data may include at least one of location information, size information, category information, and score information related to reliability or accuracy of the layout area, and may further include structural information indicating inclusion relationships, adjacency relationships, or order relationships between layout areas. This layout area data is intended to represent the overall structure of the document in a data form and can be utilized as reference data in the content extraction and processing process of a subsequent step.

[0215] Furthermore, the document comprehension unit (140) can process at least one document converted into a specific format (or a document of a specific format, a document converted into an image, an image, a document image, etc.) as input to the document analysis model (141). Here, processing at least one document (20) converted into a specific format as input to the document analysis model (141) may be performed by at least one processor.

[0216] A document analysis model (141) can analyze the structure of a document to identify at least one layout area (811 to 817, 820 and 830) each containing multiple contents having different data characteristics. Here, analyzing the structure of a document may mean a series of processing steps to recognize the arrangement and interrelationships of various contents constituting the document and to identify the logical and spatial structure of the document. For example, the process of analyzing the structure of a document may include extracting document constituent units such as multiple contents (e.g., text (plain text and formatted text, etc.), tables, charts, pictures, etc.) and determining (or identifying, judging, etc.) at least one of the location, boundary, hierarchical structure, order, and relationship of each constituent unit to model the layout structure of the entire document. This may provide structural information necessary for reconstructing the visual arrangement structure and logical flow of the document, or for content extraction and / or document editing and / or document conversion processes.

[0217] Furthermore, “identification” according to the present invention is a task of specifically distinguishing what a detected area and / or object is in a document (or an image corresponding to each page of the document), and accurately confirming (or identifying) the identity (or type, kind, etc.) of the content included in the said area. For example, recognition is generally a process of identifying the specific identity of a detected object, and in the present invention, the expression “identify” may also be referred to as “recognize.”

[0218] In one embodiment, the document analysis model (141) can identify at least one layout area (811 to 817) containing text from at least one document (20) converted into a specific format. In the present invention, the layout area containing text may also be named a “text layout area,” a “second type layout area,” a “text area,” or a “text block,” etc.

[0219] In another embodiment, the document analysis model (141) can identify at least one layout area (not shown) containing a table from at least one document (20) converted into a specific format. In the present invention, the layout area containing a table may also be named a “table layout area,” a “table area,” or a “table block,” etc.

[0220] In another embodiment, the document analysis model (141) can identify at least one layout area (820) containing an image (or picture) from at least one document (20) converted into a specific format. In the present invention, the layout area containing an image may also be named an “image (or picture) layout area,” a “first type layout area,” an “image (or picture) area,” or an “image block,” etc. In this case, the picture may be content that is recognized and processed as information different from tables and charts.

[0221] In another embodiment, the document analysis model (141) can identify at least one layout area (830) containing a chart from at least one document (20) converted into a specific format. In the present invention, the layout area containing the chart may also be named a “chart layout area,” a “chart area,” or a “chart block,” etc.

[0222] Furthermore, the document analysis model (141) can determine a label (Label, 840) corresponding to a plurality of contents included in each of at least one layout area identified from at least one document (20). For example, the document analysis model (141) can determine a label (Label, 840) indicating whether the plurality of contents included in the layout area correspond to text, an image, a table, a graph, a formula, or a figure, based on the visual features, spatial arrangement, and contextual information of the contents included in the at least one layout area.

[0223] Specifically, the document analysis model (141) can extract feature information about content included in at least one layout area constituting at least one document (20). The document analysis model (141) can determine a label (Label, 840) corresponding to the data type of each of the multiple contents included in at least one layout area by using the extracted feature information to perform inference about the data type of the content.

[0224] For example, the document analysis model (141) can calculate a reliability (or probability value) corresponding to each of the multiple data types using extracted feature information. Furthermore, the document analysis model (141) can determine a label (Label, 840) by determining one of the multiple data types as the data type of the content based on a specific criterion value related to the calculated reliability. At this time, the specific criterion value is not limited to a predefined fixed value, but can be dynamically set or adjusted according to the type of document, layout characteristics, learning stage, or purpose of document understanding. In addition, the specific criterion value is not limited to a single threshold value, but can be understood as being applicable in various forms, such as multiple criterion values, relative comparison criteria, or judgment criteria based on reliability distribution.

[0225] Additionally, the document analysis model (141) can calculate a score for each of the multiple contents (e.g., a probability value regarding whether the identified content belongs to the content of the predicted data type) based on the calculated reliability.

[0226] Furthermore, the document analysis model (141) can calculate (or determine) location information of at least one layout area constituting at least one document. Specifically, the document analysis model (141) can calculate location information of multiple contents included in each of at least one layout area in at least one document by calculating a bounding box corresponding to at least one layout area.

[0227] That is, the document analysis model (141) can receive an image of each page of at least one document as input and output at least one of a label corresponding to each of the multiple contents (e.g., text label, table label, chart label, etc.), location information of each of the multiple contents (e.g., bounding box coordinates in the format [x1, y1, x2, y2]), and a score for each of the multiple contents (e.g., a probability value indicating whether the identified content belongs to the content). The output of this document analysis model (141) can also be understood as a “document layout identification result” or a “document layout analysis result.”

[0228] To this end, it can be understood that the document analysis model (141) is trained to use a training document data set to identify at least one layout area constituting the at least one document from at least one document, and to output a label corresponding to each of the multiple contents, location information for each of the multiple contents, and a score for each of the multiple contents.

[0229] As illustrated in FIG. 8b, the document analysis model (141) according to the present invention can be trained using a training document data set (800). At this time, the training document data set (800) may include at least one of training document data (or original training document data) and augmented document data generated by the control unit (130). At this time, the augmented document data may refer to document data generated such that it includes the same layout area as the training document data, but the content included in the layout area is alternative content different from that of the training document data. The process of generating augmented document data according to the present invention will be described in detail later with reference to the relevant drawings.

[0230] A document analysis model (141) according to the present invention receives at least one document included in the training document data set (800) as input and can output layout area data (860) corresponding to a plurality of layout areas constituting the document through analysis of the input document. At this time, the layout area data (860) may include at least one of a label corresponding to each of the plurality of contents included in the at least one document, location information for each of the plurality of contents, and a score for each of the plurality of contents.

[0231] At this time, the correct layout area data (850) corresponding to the training document data set (800) can be compared with the output result of the document analysis model (141) and provided as an input to the loss function (870). Specifically, the correct layout area data (850) according to the present invention may be configured to include at least one of bounding box data representing location information corresponding to at least one layout area, type information of the layout area, category data representing the label of the content, and reading order data representing the order between layout areas.

[0232] Additionally, the correct layout area data (850) may additionally include structural information related to the size of the layout area, inclusion relationships between areas, or adjacency relationships in order to more precisely reflect the layout structure of the document, and such information may be used as reference data to evaluate the difference from the prediction result during the learning process of the document analysis model (141).

[0233] Meanwhile, in the present invention, the document analysis model (141) can be iteratively trained based on a predefined loss function (870). At this time, the loss function (870) may be configured to calculate a loss value (880) based on the difference between the layout area data (860) and the correct layout area data (850). For example, the loss function (870) may be configured to include at least one of a cross-entropy loss function, an L1 loss function, an L2 loss function, an IoU (Intersection over Union) based loss function, and a ranking loss.

[0234] Furthermore, the loss value (880) calculated during the learning process of the document analysis model (141) can be used as a feedback signal to update the parameters of the document analysis model (141) during the learning process of the document analysis model (141), and through this, the document analysis model (141) can be repeatedly trained to predict the layout structure of the document more accurately.

[0235] In the present invention, the document analysis model may be trained using a training document dataset containing augmented document data, but is not limited thereto, and may include a model trained using a training document dataset that does not contain augmented document data, or a training document dataset in which augmented document data and original document data are mixed.

[0236] Additionally, in the present invention, augmented document data may be generated and included prior to, during, or at the learning data collection stage of the document analysis model, and the document analysis model (141) may be trained based on a learning document data set configured in these various ways.

[0237] Meanwhile, in the present invention, augmented document data may be a training data component according to one embodiment for improving the performance of a document analysis model, and the configuration or training of the document analysis model is not necessarily limited by whether or not augmented document data is included.

[0238] In addition, the document analysis model (141) in the present invention is not limited to a newly trained model using augmented document data according to the present invention, but may include a model that has been retrained or fine-tuned by additionally using augmented document data in an existing trained document analysis model.

[0239] Accordingly, the description of augmented document data and the document analysis model using the same described in this specification is merely an example to aid in understanding the invention, and the learning method of the document analysis model, the composition of the learning data, and whether or not augmented document data is included may be varied within the scope of the technical concept of the invention.

[0240] In the foregoing, the document analysis model has been described in detail. Below, the process of generating augmented document data that can be used for training the document analysis model according to the present invention will be described in detail together with FIGS. 9 and FIGS. 10a to 10e. FIGS. 9 and FIGS. 10a to 10f are flowcharts and conceptual diagrams for explaining the process of generating augmented document data according to the present invention.

[0241] In the present invention, a process of generating at least one augmented document data using at least one training document data may be performed. Specifically, in the present invention, a process of specifying at least one training document data may be performed (S910, see FIG. 9).

[0242] As described above, the control unit (130) can collect at least one document to be analyzed among a plurality of documents collected (or received) from various sources (e.g., a web corpus, a document corpus, a database (DB) website, an API, a server linked to the document understanding system (100), a central server, an external server, cloud storage, a user terminal, a large dataset, etc.).

[0243] The control unit (130) according to the present invention can generate at least one augmented document data from at least one training document data using a document data generation module (131). Specifically, the control unit (130) can specify training document data for training a document analysis model (141) among at least one document. Specifically, the control unit (130) can specify training document data including annotation data (or comment data). At this time, the annotation data may include at least one of location data, size data, category data, and reading order data corresponding to each of at least one specific layout area.

[0244] As an example, in the present invention, at least one training document data including annotation data may include annotation data generated by a document analysis model, a rule-based processing module, or other automated system.

[0245] As another example, in the present invention, at least one training document data including annotation data may include annotation data that has been manually tagged or verified by a person, and may include data obtained from at least one of external document data provided with annotations already assigned and existing document data.

[0246] Accordingly, the entity generating the annotation data, the method of generation, and the timing of generation according to the present invention may be applied in various ways within the scope that does not deviate from the technical concept of the present invention.

[0247] As illustrated in FIG. 10a, the control unit (130) can specify at least one training document data (21) containing annotation data. Specifically, the control unit (130) can specify (or identify) at least one specific layout area (810a to 810i, 820a to 820d) constituting at least one training document data (21) by referring to the annotation data.

[0248] For example, the control unit (130) may specify at least one specific text layout area (810a to 810i) constituting at least one training document data (21) by referring to annotation data. As another example, the control unit (130) may specify at least one image text layout area (820a to 820d) constituting at least one training document data (21) by referring to annotation data. At this time, the control unit (130) may match reading order data (841) corresponding to each of the at least one specific layout area (810a to 810i, 820a to 820d) with the at least one specific layout area by referring to annotation data. Here, the reading order data (841) is information indicating the precedence or sequential relationship between a plurality of layout areas based on the general flow of a person reading or interpreting a document, and may represent the relative order in which each layout area must be processed or interpreted within the document.

[0249] Specifically, the reading order data (841) may include order information determined based on the spatial arrangement, positional relationship, and adjacency relationship of the layout area, and can be understood as being used as reference information to define the order of interpretation of content in various types of documents, such as text-centered documents, documents mixed with tables or images.

[0250] Next, in the present invention, a process of deactivating specific content included in at least one specific layout area constituting at least one training document data may be performed (S920, see FIG. 9).

[0251] In the process of extracting content from a document, the layout area data can be used as a criterion to identify the area within the document where content exists. For example, the content area on the document can be identified using location and size information included in the layout area data, and content data can be extracted from the identified content area. At this time, different processing logic, preprocessing methods, or postprocessing methods may be applied depending on the data characteristics of the content, and multiple models may be selectively applied to content having different data characteristics.

[0252] Specifically, the control unit (130) can disable specific content based on annotation data corresponding to at least one specific layout area and based on the layout structure of each of at least one specific layout area. In this case, the layout structure according to the present invention may refer to a set of structural characteristics including spatial characteristics such as the location, size, range, and shape of the layout area within the document, as well as relative placement relationships, inclusion relationships, or adjacency relationships with other layout areas.

[0253] In the content extraction process, layout area data and document data can be processed as inputs for multiple models corresponding to the data characteristics of the content. For example, a model corresponding to text content and a model corresponding to non-text content may be applied differently, and the outputs of each model can be integrated using a multimodal data processing method. This multimodal data processing can be performed by considering the relationships between data extracted from different types of content, and as a result, a structured document understanding result can be generated.

[0254] In this specification, the term “multimodal data processing” may refer to a method of applying different processing methods to multiple content data having different data characteristics and integrating the processing results. The multimodal data processing is not limited to simple data merging and may include data processing methods that consider the relationships, structures, or semantic associations between the content.

[0255] Specifically, the layout structure according to the present invention is information that defines the external frame or arrangement state of a layout area, and can be understood as being used as a criterion for controlling that only the content included within the layout area is selectively disabled while maintaining the layout structure.

[0256] As previously explained, “disabling content included in a layout area” according to the present invention may mean controlling so that the content of at least one layout area in the layout structure of each specific layout area is not displayed by at least one method among content masking, blurring, hiding, alpha adjustment, and color overwrite. For example, the control unit (130) may mask specific content included in at least one specific layout area constituting at least one training document data.

[0257] Specifically, the control unit (130) can extract background color information for each of at least one specific layout area using annotation data. For example, the control unit (130) can extract a color value corresponding to a specific location (e.g., the top-left point of a bounding box) included in the specific layout area based on location information for each of at least one specific layout area, and extract this as background color information for the specific layout area.

[0258] As another example, in the case of a specific layout area comprising a background composed of multiple different colors (e.g., a gradient background), the control unit (130) can extract background color information of the specific layout area by using multiple color values ​​corresponding to multiple different positions of the specific layout area. Specifically, the control unit (130) may extract multiple color values ​​corresponding to the top corner (or top corner) of the specific layout area as background color information.

[0259] As another example, the control unit (130) may extract a plurality of color values ​​corresponding to the upper, lower, left, and right corners (or edges) of the specific layout area and extract background color information for the specific layout area by interpolating the plurality of color values.

[0260] The method of extracting background color information for each of at least one specific layout area using annotation data in the present invention is not limited to the examples described above, and may include various methods of calculating background color information by sampling color values ​​at multiple locations based on the location, size, or shape of the layout area, or by combining or interpolating the sampled color values.

[0261] Furthermore, as illustrated in FIG. 10b, the control unit (130) can disable the specific content by using background color information so that the specific content area corresponding to the specific content corresponds to the background color of at least one specific layout area (810a to 810i, 820a to 820d). Specifically, the control unit (130) can control the specific content so that it is not visually identifiable by overlaying the area corresponding to the specific content with a color value corresponding to the background color information.

[0262] At this time, the control unit (130) may refer to annotation data to selectively disable only the content of specific content included in at least one specific layout area while maintaining the location, size, range, and placement relationship with other layout areas of at least one specific layout area. For example, the control unit (130) may disable only the visual representation of text, images, or other content included inside the specific layout area while maintaining the outer frame (or outline) of the specific layout area.

[0263] Additionally, the control unit (130) may process the specific content so that only the semantic information of the content is not recognized, without affecting the layout structure itself. Meanwhile, the method of maintaining the layout structure, the range of content to be deactivated, and the specific method of processing content deactivation may be varied and applied depending on the type of document, layout characteristics, or learning purpose.

[0264] Next, in the present invention, a process of generating augmented document data including alternative content having the same data characteristics as specific content in at least one specific layout area that is deactivated may be carried out (S930, see FIG. 9).

[0265] Specifically, the control unit (130) can use annotation data to identify at least one alternative content having the same data characteristics as the specific content included in each of the at least one specific layout areas among the plurality of candidate content data. For example, the control unit (130) can identify at least one alternative content corresponding to the first type content having the same data characteristics as the first type content (e.g., picture, image) included in the first type layout area (or image layout area) among the plurality of candidate content data. As another example, the control unit (130) can identify at least one alternative content corresponding to the second type content having the same data characteristics as the second type content (e.g., text) included in the second type layout area (or text layout area) among the plurality of candidate content data. At this time, the type of the at least one specific layout area can be determined based on the data characteristics of the specific content included in each of the at least one specific layout areas.

[0266] Meanwhile, it can be understood that the alternative content according to the present invention is sufficient if it is content having the same data characteristics as the specific content, and can be specified regardless of the meaning, content, subject, etc. of the specific content.

[0267] Furthermore, the control unit (130) may reflect at least one alternative content in each of at least one specific layout area based on the layout structure of each of at least one specific layout area. At this time, reflecting content in the layout area may mean placing or displaying the alternative content so that it is visually expressed within the layout area in at least one of the following ways: arranging, combining, mapping, inserting, creating, and replacing, based on the position, size, and type of the layout area corresponding to the layout area.

[0268] Meanwhile, the control unit (130) can selectively control the method of reflecting alternative content based on the inclusion relationship between the layout areas for a document in which at least one specific layout area overlaps with each other, making it difficult to distinguish the content included in each layout area. Specifically, the control unit (130) can reflect at least one alternative content in each of the at least one specific layout area based on the inclusion relationship of each of the at least one specific layout area.

[0269] As illustrated in FIG. 10c, the control unit (130) can disable a specific layout area of ​​a specific type when a specific layout area having at least one specific layout area of ​​a specific type includes a specific layout area of ​​a specific type. Here, disabling a specific layout area means maintaining the structural information of a specific layout area having one of the specific types, but controlling so that replacement content is not reflected for the specific layout area of ​​the specific type, thereby excluding the specific layout area of ​​the specific type from being subject to replacement content reflection.

[0270] For example, the control unit (130) can disable the at least one second type layout area (810j to 810m) included in the at least one first type layout area (820a) when the first type layout area (820a) of at least one specific layout area includes at least one second type layout area (810j to 810m).

[0271] Furthermore, the control unit (130) can generate augmented document data in which at least one alternative content is reflected in each of at least one specific layout area. As illustrated in FIG. 10d, the control unit (130) can generate augmented document data containing specific alternative content in at least one specific layout area (810a to 810i, 820a to 820d) constituting the training document data (21). For example, the control unit (130) can generate augmented document data (22) in which at least one first type alternative content (870a to 870i) is included in at least one text layout area (or, second type layout area, 810a to 810i). As another example, the control unit (130) can generate augmented document data (22) in which at least one second type alternative content (880a to 880d) is included in at least one image layout area (or, first type layout area, 820a to 820d).

[0272] At this time, the control unit (130) can determine the visual color of the alternative content based on the background color of at least one specific layout area and the color of the alternative content to prevent the alternative content from being sufficiently distinguished from the background and thus reducing visual identifiability.

[0273] Specifically, the control unit (130) can determine the content color corresponding to the replacement content by using background color information for each of at least one specific layout area. For example, the control unit (130) can determine the content color based on the difference between the background color of at least one specific layout area and the content color of the replacement content. Specifically, the content color according to the present invention can be determined to have a color difference from the background color according to the background color information based on a preset specific threshold value.

[0274] The control unit (130) can select (or sample) a content color to be applied to the replacement content from among at least one candidate color having a color difference greater than or equal to a preset specific threshold value from the background color according to the background color information. At this time, the preset specific threshold value is not limited to a specific value, and can be set in various ways as long as the background color of at least one specific layout area and the color of the replacement content can be visually distinguished from each other. In addition, the color difference between the background color of at least one specific layout area and the color of the replacement content according to the present invention may be defined based on color component values ​​in at least one of RGB, RGBA, HSV, HSL, Lab, Luv, or similar color spaces, or a combination thereof.

[0275] As an example, as illustrated in FIG. 10e, a control unit (130) according to the present invention may identify at least one specific layout area (810n, 810o) included in the learning document data (21) and disable specific content included in the specific layout area. At this time, the control unit (130) may refer to annotation data corresponding to each of the specific layout areas to process so that only the content included inside the specific layout area is not visually displayed while maintaining structural characteristics such as the location, size, and arrangement relationship of the specific layout area.

[0276] Furthermore, the control unit (130) may extract background color information (891) of the specific layout area (810o) prior to reflecting replacement content in the specific layout area (810o) that is disabled. Based on the background color information (891), the control unit (130) may determine the content color to be applied to the replacement content (870j) so that the replacement content (870j) can be distinguished from the background and identified. For example, the control unit (130) may select (or sample) the content color to be applied to the replacement content (870j) from one or more candidate colors that have a color difference from the background color by more than a preset threshold.

[0277] Meanwhile, as illustrated in FIG. 10f, the control unit (130) can generate augmented document data (22) by reflecting at least one noise effect on the replacement content based on a preset probability. Specifically, the control unit (130) may apply at least one of a shadow effect, a noise effect, a blur effect, a brightness change, a contrast change, an addition of a texture pattern, and a distortion effect to the replacement content (or a layout area reflecting the replacement content) only when probabilistically selected, and may apply these effects alone or in combination with each other. At this time, the preset probability is not limited to a specific probability value and can be set in various ways from at least one of the document understanding system (100) and the user terminal (10).

[0278] The noise effect according to the present invention is not limited to the examples described above and may include any effect for simulating various visual deformations that may occur in the shooting environment, scanning environment, or printing state of an actual document. Furthermore, the type, application status, intensity, frequency, and location of the noise effect may also be varied and applied within the scope of the technical concept of the present invention.

[0279] At this time, whether to apply the noise effect, the intensity of application, the location of application, and the method of application can be set in various ways depending on the type of document, layout characteristics, or learning purpose, and accordingly, it can be configured to simulate various visual variations that may occur in an actual document environment.

[0280] In the present invention, through the reflection of such probabilistic noise, the control unit (130) can generate augmented document data (22) with visual transformations added while maintaining the layout structure. As a result, while the layout structure of a specific layout area (810o) is maintained, the existing content is deactivated, and alternative content (870j) having a content color determined in correspondence with the background color (891) can be reflected in the layout area. The augmented document data (22) generated according to this method can provide document data in which the content is reconstructed in a form suitable for learning purposes while maintaining the overall layout structure and visual consistency of the document.

[0281] In the foregoing, the process of generating augmented document data that can be used for training a document analysis model (141) according to the present invention has been specifically described. Below, the process of generating output data for at least one document using the document analysis model (141) and providing it to a user terminal will be specifically described with reference to the relevant drawings. FIGS. 11a to 11e are conceptual diagrams for explaining the process of generating output data according to the present invention.

[0282] In the present invention, a process of extracting a plurality of contents included in each of at least one identified layout area may be performed (S640, see FIG. 6).

[0283] As illustrated in FIG. 11a, the document understanding unit (140) can process at least one document to be analyzed (or, document to be analyzed, 20) as input to a pre-trained document analysis model (141). Furthermore, the document analysis model (141) can output layout area data (860) corresponding to at least one document. Specifically, the document analysis model (141) can identify at least one layout area containing multiple contents from at least one document through analysis of at least one document, and output layout area data (860) corresponding to said at least one document. As previously described, the layout area data (860) may include at least one of a label corresponding to each of the multiple contents included in the at least one document, location information for each of the multiple contents, and a score for each of the multiple contents.

[0284] The content data extracted as described above can be generated as a structured document understanding result by reflecting inclusion, adjacency, or sequence relationships among the content. The structured document understanding result can be arranged according to a pre-set order or an order dynamically determined based on the processing environment, and output data can be generated through such arrangement processing. Furthermore, during the output data generation process, data filtering or selection processing may be performed to selectively reflect some or all of the content data, and the generated output data can be provided in a structured data format so that it can be used for subsequent processing by an external system or application.

[0285] In this specification, the term “structured document understanding result” may refer to result data organized such that content data extracted from a document reflects inclusion, adjacency, or sequence relationships between the contents. The structured document understanding result may be expressed as a tree structure, a graph structure, a list structure, or a combination thereof, and is not limited to a specific data format.

[0286] Furthermore, the document comprehension unit (140, or at least one processor) can extract multiple contents (160) included in each of the multiple layout areas by utilizing multiple models (150) specialized for each of the multiple contents. Specifically, the document comprehension unit (140) can process layout area data (860) and at least one document (20) as inputs to multiple models (150) corresponding to the data characteristics of each of the multiple contents, and extract multiple contents (160) included in at least one layout area from the multiple models.

[0287] For example, a plurality of models (150) for extracting a plurality of contents included in each of a plurality of layout areas can be specified, and by using (or calling) each of the specified plurality of models, a plurality of contents (160) included in each of a plurality of layout areas can be extracted.

[0288] Additionally, each of the multiple models (150) can recognize multiple contents included in each of the multiple layout areas. Alternatively, each of the multiple models can recognize multiple contents included in each of the multiple layout areas and extract the recognized contents. In this regard, the term “recognizing content” in the present invention may be understood to encompass the entire process of recognizing content and extracting the recognized content. Alternatively, the term “extracting content” in the present invention may be understood to encompass the entire process of recognizing content and extracting the recognized content. Alternatively, the process of “recognizing content” and the process of “extracting content” in the present invention may be implemented as separate processes. However, for the convenience of explanation, the following description will not be limited to any one of these methods.

[0289] The document understanding unit (140) can process at least one of the layout area data (860), which is the output of the document analysis model (141), and at least one document (20) converted into a specific format including a plurality of layout areas as input to each of the plurality of models (150).

[0290] Alternatively, the document comprehension unit (140) may call a plurality of models (150) for extracting a plurality of contents (160) contained in at least one document (20) converted into a specific format based on layout area data (860). In this case, each of the called plurality of models may extract a plurality of contents contained in each of the plurality of layout areas based on the output of the document analysis model (141).

[0291] In one embodiment, the document understanding unit (140) may call a text extraction model (142) for extracting text from each of a plurality of layout areas containing text. The text extraction model (142) may extract at least one text content contained in each of the text layout areas identified as containing text through the document analysis model (141).

[0292] In another embodiment, the document understanding unit (140) may call a table extraction model (143) for extracting a table from a layout area containing a table. The table extraction model (143) may extract at least one table content contained in a table layout area identified as containing a table through the document analysis model (141) and convert it into a specific format (e.g., “HTML”). In this case, the extraction of table content by the table extraction model (143) may be understood as extracting raw data of at least one table contained in the table layout area.

[0293] In another embodiment, the document understanding unit (140) may call a chart extraction model (144) for extracting a chart from a layout area containing a chart. The chart extraction model (144) may extract at least one chart content contained in a chart layout area identified as containing a chart through the document analysis model (141), and structure information about the entire extracted chart content into a specific format (e.g., JSON) that is pre-set. In this case, the extraction of chart content by the chart extraction model (144) may be understood as extracting raw data of the chart.

[0294] Additionally, the document understanding system (100) can process at least one document (20) converted into a specific format as input to a molecular structure formula extraction model (145) and a chemical reaction information extraction model (146), respectively, in order to identify (or recognize) and extract at least one of the molecular structure formula and chemical reaction information contained in at least one document (20) converted into a specific format.

[0295] In one embodiment, the molecular structural formula extraction model (145) can recognize a layout area containing a molecular structural formula within at least one document (20) converted into a specific format, and extract a molecular structural formula from the recognized layout area. In the present invention, the layout area containing a molecular structural formula may also be named a “molecular structural formula layout area,” a “molecular structural formula area,” or a “molecular structural formula block,” etc.

[0296] In another embodiment, the chemical reaction information extraction model (146) can recognize a layout area containing chemical reaction information consisting of reactants, reaction conditions, and products within at least one document (20) converted into a specific format, and extract chemical reaction information from the recognized layout area. In the present invention, the layout area containing chemical reaction information may also be named a “chemical reaction information layout area,” a “chemical reaction information area,” or a “chemical reaction information block,” etc.

[0297] In another embodiment, the document comprehension unit (140) processes the molecular structural formula extracted from the molecular structural formula extraction model (145) as input to the molecular structural formula conversion model (147), and the molecular structural formula conversion model (147) can convert the molecular structural formula into at least one pre-set chemical structure representation format (e.g., SMILES).

[0298] Next, in the present invention, a process of generating output data for at least one document using a plurality of contents may be carried out (S650, see FIG. 6).

[0299] In this specification, “output data” refers to data generated based on the results of structured document understanding, configured to enable further processing, analysis, or storage by an external system or application. The output data may be arranged in a pre-set order or in an order dynamically determined according to the processing environment, and some or all of the content data may be selectively reflected.

[0300] Specifically, the document comprehension unit (140) can determine (or determine) the order information of each of the multiple contents (160) extracted from a document of a specific format by using a content order sorting model (148). Specifically, the content order sorting model (148) can determine a logical reading order (e.g., ROD) corresponding to the flow of a person actually reading the document for the multiple contents (160) extracted within the document, and perform the role of sorting the content according to the determined order.

[0301] In one embodiment, the content order sorting model (148) receives layout area data (860), which is the output of the document analysis model (141), as input, analyzes the structural features of the layout included in a document of a specific format, and assigns a sequential index (or reading order) to each layout area (or each content). Through this, the content order sorting model (148) enables multiple contents (160) included in a document of a specific format to be sorted (or arranged, placed, etc.) in a semantically consistent order, and can provide a standard that is referenced when the document comprehension unit (140) subsequently generates data of a specific format (e.g., “JSON”). That is, the content order sorting model (148) restores the structural context from the document and provides information that enables the subsequent multimodal model (149) to generate various content following the flow of the original document.

[0302] The document understanding unit (140) can perform multimodal data processing for the plurality of contents having different data characteristics to generate a structured document understanding result related to the plurality of contents. Additionally, the document understanding unit (140) can provide an editing function for at least one document to the user terminal based on the structured document understanding result. Here, multimodal may refer to a technology in which artificial intelligence simultaneously understands and processes various forms of data (modalities), such as text, images, voice, and video.

[0303] In the present invention, generating a structured document understanding result related to multiple contents by performing multimodal data processing on multiple contents may mean structuring multiple contents of different modalities or formats (e.g., text, images, charts, tables, formulas, molecular structural formulas, chemical reaction information, etc.) extracted from documents through different models. Furthermore, as mentioned above, “multimodal data processing on multiple contents” refers to performing integrated processing on multiple contents corresponding to different modalities, which can also be understood as “performing integrated multimodal data processing on multiple contents having different data characteristics.”

[0304] More specifically, it may mean integratively analyzing each piece of content extracted from a document by considering its semantic associations, spatial arrangement, and logical relationships, and based on this, including (or reflecting) the document's meaning, structure (e.g., title, body text, tables, captions, relationships between each piece of content, etc.) and relationships between components, and organizing (or reconstructing) it into a structured data form (or representation) that can be interpreted by a machine (or computer).

[0305] As an example, a multimodal data integration processing process for multiple contents may include integrally understanding information between different modalities (i.e., multiple different contents) and interpreting the meaning, structure, and layout of the document to generate a document understanding result that is editable, convertible, searchable, or translated. In this case, the multimodal data integration processing process may include a process of combining the recognition results of different modalities into a single integrated expression by mutually merging (or aligning) them based on positional relationships, semantic associations, and document context, rather than simply merging them. In this case, the process may include mapping text and images corresponding to the same area, establishing correspondence relationships between table and chart images and corresponding raw data, and analyzing semantic connection relationships between body text, titles, comments, captions, shapes, and formulas.

[0306] Alternatively, the multimodal data integration processing process for multiple contents may include a processing process that processes (or interprets) contents having different data characteristics through a pre-configured processing technique (or method) and generates a structured document understanding result by reflecting the semantic and structural relationships between each content. Here, the pre-configured processing technique may include a multimodal data integration processing technique and / or a multimodal analysis technique. In one embodiment, the pre-configured processing technique may include at least one of: i) a technique for normalizing features by vectorizing or embedding extracted information and mapping them to a common representation space; ii) a technique for matching and / or inferring relationships between contents based on the spatial arrangement or semantic similarity of the contents; iii) a technique for fusing features of different modalities using an Early fusion and / or Late fusion and / or Hybrid fusion method; and v) a technique for aligning semantic correspondence relationships between contents having different data characteristics.

[0307] Specifically, the multimodal model (149) can convert a document of a specific format into an editable form based on at least one of the order information specified through the content order sorting model (148), layout area data (860), and multiple contents (160) extracted from each of the multiple models (150). Here, a document converted into an editable form may mean a document that has been reconstructed to allow modification, addition, deletion, etc., while maintaining the meaning and structure of the multiple contents included in the document. That is, it may mean a document that has been reconstructed to allow for free modification and editing in various document formats (e.g., PPT, Excel, Word, etc.) by recognizing and extracting each content constituting the document from a document in which modification and / or structural editing of the contents is restricted.

[0308] In this case, the document comprehension unit (140) may configure a prompt including at least one of the order information specified through the content order sorting model (148), layout area data (860), multiple contents (160) extracted from each of the multiple models (150), and user request information received from the user terminal (10), and input the prompt into the multimodal model (149). For example, the document comprehension unit (140) may further include a prompt generation unit that generates a prompt to be input into the multimodal model (149). As an example, the prompt generation unit may generate a prompt requesting output data (30) for at least one document (20) based on at least one of the order information specified through the content order sorting model (148), layout area data (860), multiple contents (160) extracted from each of the multiple models (150), and user request information received from the user terminal (10).

[0309] As an example, the prompt generation unit according to the present invention may include at least one of a large language model based on T5 (Text-to-Text Transfer Transformer), BART (Bidirectional and Auto-Regressive Transformer), GPT (Generative Pre-trained Transformer), or LLaMA (Language Model for Many Applications), a rule-based template matching algorithm, a conditional prompting technique, a contextual embedding selection module, or a few-shot prompt generator. In this case, the model or algorithm included in the prompt generation unit according to the present invention is not limited to the examples described above and may further include artificial intelligence models and algorithms that perform the same function.

[0310] In the present invention, the document comprehension unit (140) is described as including a prompt generation unit, but is not limited thereto. If the prompt generation unit exists on an external server, the document comprehension unit (140) may call or control the functions of the prompt generation unit by interacting with the external server. Furthermore, the multimodal model (149) according to the present invention can convert a document of a specific format into an editable form using (or based on) the input prompt.

[0311] The document comprehension unit (140) can integrate layout area data (860) with the order information of each of the multiple contents specified through the content order sorting model (148) and input the integrated data into the multimodal model (149). The multimodal model (149) can convert a document of a specific format into an editable form using the input integrated data.

[0312] Alternatively, the document understanding system (100) may input data of a specific format (e.g., JSON) into the multimodal model (149). The multimodal model (149) may use the input data of the specific format to convert a document of the specific format into an editable form.

[0313] In another embodiment, the multimodal model (149) can i) analyze the meaning of each of the multiple contents to summarize the content of the document, ii) reconstruct sentences, or iii) extract key keywords. Additionally, the multimodal model (149) can recognize and connect reference relationships between text, images, charts, and tables among the extracted multiple contents, and perform structural and descriptive transformations necessary for an output form (e.g., file format) corresponding to a user request. Furthermore, the multimodal model (149) can generate natural narratives that match the context or style of the document, and, if necessary, perform additional semantic judgments for visual editing, such as changing the background or highlighting elements, through image-based context analysis.

[0314] In the present invention, a process of providing output data to a user terminal may be performed (S660, see FIG. 6).

[0315] Furthermore, the document comprehension unit (140) can provide output data (30) output from the multimodal model (149) to the user terminal (10). Specifically, the document comprehension unit (140) can provide output data (30) output from at least one document (20) to the user terminal (10) through at least one page (2000) for providing output data (30).

[0316] Specifically, at least one of the control unit (130) and the document understanding unit (140) can receive a user request (or user input) including at least one document from a user terminal (10) through at least one page (2000).

[0317] In the following description, the control unit (130) according to the present invention is described as performing the process of providing output data to a user terminal; however, this is merely an embodiment performed in conjunction with the components of the document understanding system (100) and the document understanding unit (140), and the process of generating, processing, or providing output data is not necessarily limited to the control unit (130). That is, the provision of the output data may be performed independently or cooperatively by at least one of the document understanding unit (140), the communication unit, or other components constituting the document understanding system (100), and it can be understood that the entity performing the task may be varied depending on the implementation form of the system, the service structure, or the operating environment.

[0318] For example, as illustrated in FIG. 11b, the control unit (130) can activate the graphic object (2003) based on the selection of a graphic object (e.g., “convert / create”, 2003) associated with a document conversion (or creation) request included in at least one page (2000) from the user terminal (10). Then, with the graphic object (2003) activated, the control unit (130) can receive a user request (2150) including at least one document (e.g., “Document A_20251130”, 2100) based on the selection of a graphic object (e.g., “confirm”, 2150a) associated with a document and user request reception function from the user terminal (10). Here, the user request (2150) may further include user input (e.g., “file format”, 2101) for converting the form of the document (2100) to be converted into at least one specific form desired by the user among a plurality of forms. In this case, receiving a user request including at least one document may also be understood as receiving together the document and the user request (or user input, user selection, etc.) for converting the document (2100) into a form that meets the user’s needs (or is desired by the user). Additionally, the specific form corresponds to the user request and may mean at least one form different from the form of at least one document (for example, if the form of the original document is the first form, the form of the converted document is the second form).

[0319] Subsequently, the control unit (130) can check the extension of the document (2100) and convert the document (2100) into a specific format that has been pre-set. Then, the control unit (130) can identify multiple layout areas each containing multiple contents having different data characteristics through analysis of the document converted into the specific format, and can extract multiple contents included in each of the multiple layout areas by using multiple models specialized for each of the multiple contents. Since more specific details regarding this have been described above, they will be omitted below to avoid duplication of explanation.

[0320] Furthermore, the control unit (130) can use multiple contents extracted from a document converted into a specific format to convert the document converted into a specific format into a form corresponding to a user request (2150), and as a response to the user request (2150), provide the document converted into a form corresponding to the user request to the user terminal (10). Here, the term "document converted to correspond to the user request (2150)" may include a document converted into a specific form corresponding to the user request.

[0321] More specifically, the control unit (130) can understand the relationship between the flow of the document and the extracted multiple contents, and convert the form of the document (or the document converted into a specific format) into a specific form corresponding to the user request. Here, understanding the relationship between the flow of the document and the extracted multiple contents may include the process of identifying the logical connection structure between paragraphs in the document and analyzing the technical and explanatory purpose conveyed by each paragraph. It may also include the process of identifying the interrelationships regarding how images, charts, tables, etc. included in the document complement or explain the text. Through this, the correspondence between the concepts, steps, and components presented by the text and the visual information provided by the images can be structurally understood. Furthermore, the logical consistency of the entire document can be ensured by automatically determining whether an image supports the explanation of a specific paragraph or assists the flow of the technical composition. Based on this analysis, text-image mapping, the reorganization of the information order, and the rearrangement of visual elements become possible, and distortion of the original text's meaning can be prevented during the document conversion and / or creation process.

[0322] Furthermore, converting into a form corresponding to (or corresponding to) a user request may mean converting the structure and / or format (or form, style) of a document based on (or criterion for) the content of the user's request. This may include a process of generating a document in a form that satisfies the user's intended purpose and requirements by reconstructing the document to correspond to the content and format of the user's request using multiple contents extracted from the document.

[0323] Alternatively, converting into a form that corresponds to a user request involves analyzing the user's request based on multiple contents extracted from the document and rearranging and / or converting the content in a manner corresponding to the conditions of the request. Through this, by satisfying the level of information provision and presentation format requested by the user, a document that satisfies the user's request can be provided.

[0324] In other words, “response” can be understood as organizing content (e.g., selecting, combining, rearranging, etc.) to correspond to the intent, purpose, requirements, and output format included in the user request, and “satisfaction” as generating a document that satisfies the scope of information and presentation method requested by the user.

[0325] For example, let us assume that a user request (2150) to convert a document converted to a specific format into a PPT is received. As illustrated in (a) of FIG. 11c, the control unit (130) can convert the document converted to a specific format into a first form (e.g., PPT) corresponding to the user request (2150) and provide the converted document (2100a) to the user terminal (10).

[0326] As another example, let us assume that a user request (2150) is received to convert a document converted into a specific format into the form of a blog post. As illustrated in (b) of FIG. 11c, the control unit (130) can convert the document converted into a specific format into a second form (e.g., a blog post) corresponding to the user request (2150) and provide the converted document (2100b) to the user terminal (10).

[0327] Let us assume, as another example, that a user request (2150) is received to convert a document converted into a specific format into a poster form. As illustrated in (c) of FIG. 11c, the document understanding system (100) can convert the document (2100) into a third form (e.g., a poster) corresponding to the user request (2150) and provide the converted document (2100c) to the user terminal (10).

[0328] Meanwhile, the control unit (130) according to the present invention can convert the form of a document into a form desired by the user by reflecting style information. More specifically, the control unit (130) can convert the form of a document into a form desired by the user by reflecting the user's preferred style information and provide it to the user.

[0329] For example, as illustrated in FIG. 11d, the control unit (130) can activate the graphic object (2003) based on the selection of a graphic object (e.g., “conversion / creation”, 2003) associated with a document conversion (or creation) request included in at least one page (2000) from the user terminal (10). Then, with the graphic object (2003) activated, the control unit (130) can receive a user request (2350) including at least one document (e.g., “Document B_20251201”, 2300) based on the selection of a graphic object (2350a) associated with a document and user request receiving function from the user terminal (10). At this time, the user request (2350) may further include at least one of style information (2301) to be reflected in the document (2300) to be converted, paragraph information (2302), and user input (e.g., “file format”, 2303) for converting the document (2300) into a form desired by the user.

[0330] However, the style information (2301) and / or paragraph information (2302) may be information that is automatically reflected according to the context of the document entered by the user, or information that is automatically reflected based on (or derived from) user history information. In one embodiment, the user history information may include style information and / or paragraph information that the user frequently uses or prefers.

[0331] Subsequently, the control unit (130) can check the extension of the document (2300) and convert the document (2300) into a specific format that has been pre-set. Then, the control unit (130) can identify multiple layout areas each containing multiple contents having different data characteristics through analysis of the document converted into the specific format, and can extract multiple contents included in each of the multiple layout areas by using multiple models specialized for each of the multiple contents. Since more specific details regarding this have been described above, they will be omitted below to avoid duplication of explanation.

[0332] Furthermore, the control unit (130) can use multiple contents extracted from a document converted into a specific format to convert the document converted into a specific format into a form corresponding to a user request (2350), and as a response to the user request (2350), provide the document converted into a form corresponding to the user request to the user terminal (10). At this time, the control unit (130) can convert the form of the document converted into a specific format into a form corresponding to the user request by reflecting the style information based on the fact that the user request (2350) includes style information.

[0333] In this case, the document comprehension unit (140) can determine the order information of each of the multiple contents extracted from a document converted into a specific format by using a content order sorting model (148). For example, as shown in FIG. 11e, the content order sorting model (148) can determine the order information of each of the multiple contents extracted from a document (2300a) converted into a specific format and sort the contents according to the determined order information.

[0334] At this time, at least some of the multiple contents being aligned may correspond to content that has a style corresponding to user input reflected through the style module (575). More specifically, the control unit (130) may reflect (or apply) a style corresponding to user input to at least some of the multiple contents extracted from a document converted to a specific format based on the inclusion of style information in the user request (2350). For example, the control unit (130) may reflect at least one of the style information (2301) and paragraph information (2302) included in the user request (2350) to at least some of the multiple contents (e.g., text) extracted from a document converted to a specific format.

[0335] Finally, the control unit (130) can provide the user terminal (10) with a document converted to correspond to the user request (2350). Here, the term "document converted to correspond to the user request (2350)" may include a document that reflects at least one of the style information and paragraph information entered by the user and is converted into a specific form according to the user request (2350).

[0336] For example, let us assume that a user request (2350) to convert a document into PPT is received. As illustrated in (a) to (d) of FIG. 11e, the control unit (130) may provide the converted document to the user terminal (10), which includes a plurality of slides (2401, 2402, 2403, 2404) and reflects at least one of the style information and paragraph information entered by the user. At this time, the plurality of slides (2401, 2402, 2403, 2404) may include slides created by arranging a plurality of contents according to the order information specified from the content order sorting model (148) in the multimodal model (149).

[0337] As described above, the document understanding method and system according to the present invention can increase (or enhance) user convenience by recognizing the structure of various types of documents and the relationships between multiple contents (e.g., text, tables, graphs, charts, molecular structural formulas, etc.) included in the documents, thereby automatically extracting accurate information. Through this, users can minimize the time required to find necessary information in documents and reduce inconvenience.

[0338] Furthermore, according to the document understanding method and system of the present invention, a plurality of contents included in a document can be identified individually, and a plurality of contents can be automatically extracted from the document by using a plurality of models specialized for each of the plurality of contents having different data characteristics. Through this, the user can receive the necessary information quickly and accurately.

[0339] Furthermore, the document understanding method and system according to the present invention can prevent information omission and enhance the sophistication of analysis by deeply analyzing visual information of various forms of content included in a document. Through this, the present invention can accurately process a vast amount of documents, thereby improving the user's overall work or business processing speed.

[0340] Furthermore, according to the document understanding method and system of the present invention, by deeply analyzing various forms of content included in a document and providing a solution optimized for the user based on this, the efficiency of document processing can be maximized and the user's overall work efficiency can be improved.

[0341] Furthermore, according to the document understanding method and system of the present invention, it is possible to understand text, graphs, and tables contained in a document, while simultaneously interpreting complex molecular structural formulas and chemical reaction information. Through this, the present invention can enhance the research efficiency of chemists and support innovative discoveries in the field of chemistry. In other words, the present invention can satisfy the needs of the research field through a customized model that understands even information related to the chemical domain within the document.

[0342] Furthermore, according to the document understanding method and system of the present invention, molecular structural formulas can be accurately recognized in documents using a model specialized for molecular structural formulas, and a large-scale database of molecular structural formulas can be constructed based on this. Through this, researchers can utilize the database to search for chemical structural formulas and efficiently carry out large-scale analysis tasks required for research, thereby reducing the time required for research and / or development.

[0343] Furthermore, according to the document understanding method and system of the present invention, a plurality of contents extracted from a document using a plurality of models and output data generated based on said plurality of contents can be visualized and provided through a user interface. Through this, the user can intuitively recognize the information needed and understand it more quickly, thereby improving the efficiency of work and / or research.

[0344] Furthermore, according to the document understanding method and system of the present invention, augmented document data having the same layout structure as the actual training document data used for training a Document Layout Analysis (DLA) model can be generated. Through this, large-scale training data reflecting various layout characteristics and visual variations of actual documents can be efficiently secured, and the generalization performance and document understanding accuracy of the trained document analysis model can be effectively improved.

[0345] Each processing step of the document understanding method described above is not necessarily limited to the order specified above; depending on the processing environment or implementation method, it may be performed sequentially, or some steps may be performed in parallel or repeatedly. This flexibility in the execution of steps enables efficient document analysis and content processing depending on the type, size, or complexity of the document.

[0346] Meanwhile, the present invention described above can be implemented based on a quantum computer. The present invention implemented based on a quantum computer may include a qubit-based quantum processor and quantum memory, and may include software and hardware interfaces optimized for quantum computation.

[0347] Quantum processors in quantum computers utilize qubits to efficiently process complex operations through parallel computation, quantum entanglement, and quantum superposition, which cannot be performed by the binary bits of classical computers. Quantum processors process data using quantum gates and can provide exponential speed improvements for specific problems.

[0348] Meanwhile, the present invention described above can be implemented as a program that is executed by one or more processes on a computer and can be stored on a computer-readable medium (or recording medium).

[0349] Furthermore, the present invention described above can be implemented as computer-readable code or instructions on a medium on which a program is recorded. That is, the present invention can be provided in the form of a program.

[0350] Meanwhile, computer-readable media include all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include HDD (Hard Disk Drive), SSD (Solid State Disk), SSD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0351] Furthermore, the computer-readable medium may be a server or cloud storage that includes a storage and is accessible to an electronic device via communication. In this case, the computer may download the program according to the present invention from the server or cloud storage via wired or wireless communication.

[0352] A computer program may reach the system (100) through various suitable transmission mechanisms. The transmission mechanism may be, for example, a computer-readable storage medium, a computer program product, a memory device, a recording medium such as a CD-ROM or DVD, or a product that tangibly embodies the computer program. The transmission mechanism may be a signal configured to reliably transmit the computer program through air or an electrical connection. The system (100) may propagate or transmit the computer program as a computer data signal.

[0353] Furthermore, references to 'computer-readable storage media,' 'computer program products,' 'computer programs embodied in a tangible form,' etc., or to 'controller,' 'computer,' 'processor,' etc., should be understood to include not only computers with various architectures such as single / multi-processor architectures and sequential (Von Neumann) / parallel architectures, but also specialized circuits such as Field-Programmable Gate Arrays (FPGAs), Application Specific Circuits (ASICs), signal processing units, and other devices. References to computer programs, instructions, code, etc., should be understood to include software for programmable processors or firmware, such as programmable content for hardware devices, whether it is instructions for a processor or configuration settings for a fixed-function device, gate array, or programmable logic device.

[0354] Furthermore, in the present invention, the computer described above is an electronic device equipped with a processor, namely a CPU (Central Processing Unit), and no special limitations are placed on its type.

[0355] Meanwhile, the above detailed description should not be interpreted restrictively in all respects but should be considered exemplary. The scope of the invention shall be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the invention are included within the scope of the invention.

Claims

1. Regarding methods performed by a computer, A step of identifying at least one document to be analyzed; A step of analyzing the document using a document analysis model and identifying at least one layout area having different attributes from the document; A step of extracting at least one content included in each of the identified layout areas; and A document understanding method characterized by including the step of generating output data for the document using at least one extracted content.

2. In Paragraph 1, A document understanding method characterized in that the above document analysis model is a model trained using a training document dataset that includes augmented document data generated based on at least one training document data.

3. In Paragraph 1, The above document analysis model is, A document understanding method characterized by identifying the layout area containing at least one content from the document through analysis of the document, and generating layout area data corresponding to the document.

4. In Paragraph 3, The above layout area data is, A document understanding method characterized by including at least one of a label, location information, size information, and score corresponding to each of the contents included in the above document.

5. In Paragraph 4, In the step of extracting the above content, A document understanding method characterized by processing the above layout area data and the above document as inputs for a plurality of models corresponding to the data characteristics of each of the above content, and extracting content data included in the above layout area based on the output of the plurality of models.

6. In Paragraph 5, In the step of generating the above output data, Based on the above layout area data, multimodal data processing is performed on the content data output from the plurality of models to generate a structured document understanding result, and A document understanding method characterized by generating output data by arranging the above-mentioned structured document understanding results.

7. In Paragraph 6, A document understanding method characterized by the above multimodal data processing including a data processing method that combines text-based processing and non-text-based processing.

8. In Paragraph 6, A document understanding method characterized in that the above-mentioned structured document understanding result is generated to reflect sequential or hierarchical relationships between contents.

9. In Paragraph 5, A document understanding method characterized by the above plurality of models including models corresponding to different content types.

10. In Paragraph 5, A document understanding method characterized in that the step of extracting the above content data includes different preprocessing or postprocessing data processing depending on the data characteristics of the content.

11. In Paragraph 4, A document understanding method characterized in that the above layout area data further includes structural information indicating inclusion or adjacency relationships between the above layout areas.

12. In Paragraph 4, A document understanding method characterized in that the above layout area data further includes score information related to the reliability or accuracy of the above document analysis model.

13. In Paragraph 1, The above content includes at least one of text, tables, charts, molecular structural formulas, or chemical reaction information, and A document understanding method characterized in that each of the above-mentioned contents is included in a layout area corresponding to the data characteristics of each of the above-mentioned contents.

14. Memory for storing instructions; and It includes at least one processor electrically connected to the memory, and When the above instructions are executed by the at least one processor, the at least one processor, Identify at least one document to be analyzed, and Analyze the above document using a document analysis model, and identify at least one layout area having different attributes from the above document, and Extract at least one content included in each of the identified layout areas, and A document understanding system characterized by generating output data for the document using at least one extracted content.

15. In a method performed by a computer, A step of specifying at least one training document data; A step of disabling specific content included in at least one specific layout area constituting the above at least one training document data; A step of generating augmented document data including alternative content having the same data characteristics as the specific content in at least one specific layout area that is deactivated; and A method for training a document analysis model characterized by including the step of training a document analysis model using a training document dataset containing the augmented document data above.

16. In Paragraph 15, The step of disabling the specific content mentioned above is, A method for learning a document analysis model characterized by being performed based on annotation data corresponding to at least one specific layout area, and based on the layout structure of each of the at least one specific layout area.

17. In Paragraph 16, The step of disabling the specific content mentioned above is, A step of extracting background color information for each of the at least one specific layout area using annotation data; and A method for learning a document analysis model characterized by including the step of deactivating the specific content using the background color information above so that the specific content area corresponding to the specific content corresponds to the background color of the specific layout area.

18. In Paragraph 16, The step of generating the above-mentioned augmented document data is, A step of identifying, among multiple candidate content data using annotation data, alternative content having the same data characteristics as specific content included in each of the specific layout areas; A step of reflecting at least one alternative content in each of the specific layout areas based on the above layout structure; and A method for training a document analysis model characterized by including the step of generating augmented document data that reflects at least one alternative content.

19. In Paragraph 18, The step of reflecting the above alternative content is, A method for learning a document analysis model characterized by being performed based on the inclusion relationship of each of the above at least one specific layout area.

Images