Image information extraction method and apparatus, device, and storage medium

Abstract

An image information extraction method and apparatus, a device, and a storage medium. The method is applied to a first device, and comprises: when a first image comprises a target object, performing image enhancement on the first image to obtain an enhanced second image (101); determining one or more target objects comprised in the second image and object type(s) thereof (102); determining an extraction model for the target object on the basis of the object type of the target object (103); using the extraction model for the target object to perform image information extraction on the target object to obtain association information between attribute information of multiple elements of the target object and the elements (104); and converting the association information between the attribute information of the multiple elements of the target object and the elements into a target file format, so as to obtain a target file of the target object (105). In this way, a mobile terminal can quickly and accurately extract the attribute information and association information of elements in each object from an image, and convert same into a standardized target file format, thereby improving the readability and versatility of files. Furthermore, secondary editing of a target object can be implemented on the basis of a target file.

Description

A method, apparatus, device and storage medium for extracting image information

[0001] Cross-references to related applications

[0002] This application is based on and claims priority to a prior Chinese patent application, application number 202411642405.5, filed on November 15, 2024, entitled "A Method, Apparatus, Device and Storage Medium for Image Information Extraction", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of computer technology, and in particular to an image information extraction method, apparatus, device and storage medium. Background Technology

[0004] Images can intuitively display data and information, making it easier to understand and analyze, and play an important role in various fields. However, due to limitations in image formats, the data and information displayed cannot be directly extracted from images. Therefore, related technologies typically employ multi-stage processing methods to extract information from images. First, key elements are detected from the image. Then, based on a series of post-processing rules, the detected key elements are further processed and analyzed to obtain the relevant information displayed in the image.

[0005] However, existing image information extraction methods have low extraction accuracy and poor versatility, making them unable to cope with complex and ever-changing real-world environments. Summary of the Invention

[0006] This application aims to provide an image information extraction method, apparatus, device, and storage medium to improve the accuracy and versatility of image information extraction.

[0007] The technical solution of this application is implemented as follows:

[0008] Firstly, an image information extraction method is provided, including:

[0009] If the first image to be processed contains the target object, the first image is enhanced to obtain the enhanced second image;

[0010] Determine one or more target objects and their object types included in the second image;

[0011] Based on the object type of the target object, determine the extraction model of the target object;

[0012] The target object is extracted using the extraction model of the target object to obtain the attribute information of multiple elements of the target object and the association information between the elements;

[0013] The attribute information of the multiple elements of the target object and the association information between the elements are converted into a target file format to obtain the target file of the target object.

[0014] Secondly, an image information extraction device is provided, comprising:

[0015] The image enhancement unit is configured to enhance the first image when the first image to be processed includes a target object, thereby obtaining an enhanced second image;

[0016] The determining unit is configured to determine one or more target objects and their object types included in the second image; and to determine the extraction model of the target objects based on the object types of the target objects.

[0017] The extraction unit is configured to extract image information of the target object using the extraction model of the target object, and obtain attribute information of multiple elements of the target object and the association information between the elements;

[0018] The extraction unit is further configured to convert the attribute information of the plurality of elements of the target object and the association information between the elements into a target file format to obtain the target file of the target object.

[0019] Thirdly, a first device is provided, comprising: a processor and a memory configured to store a computer program capable of running on the processor.

[0020] Wherein, the processor is configured to execute the steps of the aforementioned method when running the computer program.

[0021] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the aforementioned method.

[0022] Fifthly, a computer program product includes a computer program that, when executed by a processor, implements the steps of the aforementioned method.

[0023] This application provides an image information extraction method, apparatus, device, and storage medium, enabling mobile devices to quickly and accurately extract attribute and association information of elements in various objects from an image, and convert it into a standardized target file format, improving the readability and versatility of the file. Furthermore, it also allows for secondary editing of the target object based on the target file. Attached Figure Description

[0024] Figure 1 is a schematic diagram of the first process of the image information extraction method in an embodiment of this application;

[0025] Figure 2 is a schematic diagram of the single-image information extraction interface in an embodiment of this application;

[0026] Figure 3 is a schematic diagram of the single-image information extraction interface in an embodiment of this application;

[0027] Figure 4 is a schematic diagram of the multi-image information extraction interface in an embodiment of this application;

[0028] Figure 5 is a schematic diagram of the multi-image information extraction interface in an embodiment of this application;

[0029] Figure 6 is a schematic diagram of the automatic chart information extraction interface in an embodiment of this application;

[0030] Figure 7 is a schematic diagram of the automatic chart information extraction interface in an embodiment of this application;

[0031] Figure 8 is a schematic diagram of the interface for copying and sharing the extracted chart information results in an embodiment of this application.

[0032] Figure 9 is a schematic diagram of the editing interface of the chart data source in the embodiment of this application;

[0033] Figure 10 is a schematic diagram of the editing interface for chart types and chart elements in an embodiment of this application;

[0034] Figure 11 is a schematic diagram of the mind map editing interface in an embodiment of this application;

[0035] Figure 12 is a schematic diagram of the image information extraction interface in the "scan" function according to an embodiment of this application;

[0036] Figure 13 is a schematic diagram of the second process of the image information extraction method in an embodiment of this application;

[0037] Figure 14 is a schematic diagram of image samples in an embodiment of this application;

[0038] Figure 15 is a schematic diagram of the training process of the image pre-classification module in an embodiment of this application;

[0039] Figure 16 is a flowchart of the fine-tuning process of the image pre-classification module in an embodiment of this application;

[0040] Figure 17 is a schematic diagram of the image quality enhancement process in an embodiment of this application;

[0041] Figure 18 is a comparative schematic diagram of trapezoidal correction before and after in an embodiment of this application;

[0042] Figure 19 is a schematic diagram of the trapezoidal correction process in an embodiment of this application;

[0043] Figure 20 is a schematic diagram of the chart information extraction process in an embodiment of this application;

[0044] Figure 21 is a mind map annotation diagram in an embodiment of this application;

[0045] Figure 22 is a flowchart illustration of an embodiment of this application;

[0046] Figure 23 is a schematic diagram showing the detection boxes for the center point and intersection of the pie chart in an embodiment of this application;

[0047] Figure 24 is a schematic diagram of the composition structure of the image information extraction device in an embodiment of this application;

[0048] Figure 25 is a schematic diagram of the composition structure of the first device in the embodiment of this application. Detailed Implementation

[0049] In order to gain a more detailed understanding of the features and technical content of the embodiments of this application, the implementation of the embodiments of this application will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference and illustration only and are not intended to limit the embodiments of this application.

[0050] Figure 1 is a schematic diagram of the first process of the image information extraction method in an embodiment of this application. As shown in Figure 1, the method may specifically include:

[0051] Step 101: If the first image to be processed includes the target object, perform image enhancement on the first image to obtain the enhanced second image;

[0052] The first image is the image to be processed. Specifically, the first image can be an image captured by a camera, the currently displayed image, or an image determined through an image selection operation.

[0053] Image enhancement improves image quality, including sharpness, color reproduction, and detail. By enhancing edge information, contour information, and contrast, the differences between features of different objects in an image become more pronounced, thereby improving image resolution and detail. This helps users more accurately identify and distinguish different objects and features in an image. Enhanced images can more effectively extract and recognize feature information, improving the accuracy and efficiency of subsequent image information extraction.

[0054] In some embodiments, image enhancement is performed on a first image to obtain an enhanced second image, including: performing image quality analysis on the first image to determine one or more image enhancement algorithms corresponding to the first image; and performing image enhancement processing on the first image based on one or more image enhancement algorithms to obtain an enhanced second image.

[0055] Image quality analysis is used to analyze image quality and select appropriate image enhancement algorithms based on the actual quality of the image to improve image quality and thus enhance the accuracy of subsequent image information extraction. These image enhancement algorithms include, but are not limited to, at least one of the following: trapezoidal correction, surface correction, moiré removal, and deblurring.

[0056] For example, the input image is subjected to trapezoidal correction using a trapezoidal correction model, surface correction using a surface correction model, moiré removal using a demoiré model, and deblurring using a deblurring model.

[0057] In some embodiments, the enhancement model performs image enhancement processing based on a corresponding image enhancement algorithm. For example, the trapezoidal correction model is configured to implement the following steps: acquiring the original document image to be corrected; predicting document edge probabilities to obtain document edges; fitting straight lines to the document edges to obtain a set of lines; locating document vertices based on the set of lines to obtain the coordinates of the document's four vertices; obtaining the coordinates of four target vertices of the document in the target plane based on the coordinates of the document's four vertices; calculating the perspective transformation matrix M based on the coordinates of the four vertices and the coordinates of the four target vertices; performing a padding operation on the image to be corrected based on the coordinates of the four vertices; and then performing a perspective transformation on the padded image to be corrected based on M to complete document correction.

[0058] In some embodiments, the enhancement model is a deep neural network model. This model implements one image enhancement process, or multiple image enhancement methods.

[0059] When the augmentation model is a deep neural network model, the method further includes: obtaining a third training sample set, which includes third image samples and image ground truth; constructing an initial augmentation model; and training the initial augmentation model of the target object using the third training sample set to obtain a trained augmentation model.

[0060] For example, the deblurring model is a deep neural network model, and the training process of the deblurring model can be as follows:

[0061] S1: Collect clear image data and construct dataset 1;

[0062] S2: Add random noise to the clear image to obtain dataset 2.

[0063] S3: Add motion blur to the noisy image to obtain dataset 3;

[0064] S4: Add tadpole-shaped deformation to the blurred image to obtain dataset 4;

[0065] S5: Train the deblurring model using dataset 1 and dataset 4 to obtain the trained deblurring model.

[0066] In some embodiments, the method further includes: if the first image does not contain the target object, determining that the first image is an unextractable image, and skipping the steps of the image information extraction method. By identifying unextractable images, the applicability of the image information extraction method provided in this application embodiment is expanded, and processing efficiency is improved.

[0067] In some embodiments, the method further includes: if the first image does not include the target object, outputting a first prompt message to inform the user that the first image is an unextractable image. By outputting the first prompt message, the user is promptly informed that the current image is a rejected image, thus improving the user experience.

[0068] In some embodiments, the method further includes: pre-classifying the first image to determine whether the first image includes a target object. For example, the first image is pre-classified using an image pre-classification model to obtain a classification result, which indicates whether the first image includes a target object. The image recognition model includes a feature extraction network and a classification head. The feature extraction network is configured to extract features from the first image to obtain a feature embedding representation, and the classification head is configured to perform image classification based on the feature embedding representation to determine whether the first image is a target object.

[0069] Pre-classification is used to determine whether the image type of the first image is one that supports image information extraction operations, that is, whether the image type includes an extractable target object; if it does, the subsequent image information extraction steps are continued; if it does not, the first image can be determined as a rejected image and ignored to avoid wasting computing power.

[0070] In some embodiments, the method further includes: obtaining an original training sample set, the original training sample set including original image samples and their category labels, the category labels being used to indicate whether the image samples contain a target object; performing data augmentation on the original image samples in the original training sample set to obtain an augmented first training sample set; and training a first image pre-classification model using the first training sample set to obtain a trained image pre-classification model.

[0071] For example, data augmentation may include: performing random data augmentation on image samples based on one or more random data augmentation strategies to obtain augmented image samples; performing data mixing operations on the augmented image samples to obtain mixed image samples; and constructing a first training sample set using the augmented image samples and the mixed image samples.

[0072] For example, training a first image pre-classification model using a first training sample set to obtain a trained image pre-classification model may include: inputting augmented image samples into the first image pre-classification model to obtain an augmented image feature embedding representation output by the feature extraction network of the first image pre-classification model and a classification result output by the classification head; inputting mixed image samples into the first image pre-classification model to obtain a mixed image feature embedding representation output by the feature extraction network of the first image pre-classification model; calculating a first loss function using the classification result and class label; adding gradient perturbation to the augmented image feature embedding representation based on the first loss function to obtain a perturbed image feature embedding representation; calculating a second loss function based on the perturbed image feature embedding representation, the augmented image feature embedding representation, and the mixed image feature embedding representation; and optimizing the weight parameters of the image pre-classification model based on the second loss function to obtain a second image pre-classification model.

[0073] Feature embedding representation generally refers to the process of converting high-dimensional discrete features (such as pixel values, colors, and textures in an image) into low-dimensional continuous feature vectors. Feature embedding representation can be understood as image feature data; this representation method aims to capture key information in an image while reducing the dimensionality of the data to facilitate subsequent tasks such as machine learning, deep learning, or information retrieval.

[0074] In some embodiments, the second loss function includes: the Info NCE (Information Noise-Contrastive Estimation) loss function and the SupCon (Supervised Contrastive Learning) loss function. The Info NCE loss function is a commonly used loss function in contrastive learning. Its core idea is to learn meaningful representations by maximizing the similarity of similar sample pairs while minimizing the similarity of dissimilar sample pairs. The SupCon loss function bridges the gap between self-supervised learning and fully supervised learning, and enables contrastive learning to be applied to supervised environments.

[0075] In some embodiments, the method further includes: freezing the weight parameters of the feature extraction network of the second image pre-classification model, optimizing the weight parameters of the classification head of the second image pre-classification model, and obtaining a trained image pre-classification model; or, replacing the classification head of the second image pre-classification model with a new classification head; freezing the weight parameters of the feature extraction network of the second image pre-classification model, optimizing the weight parameters of the new classification head of the second image pre-classification model, and obtaining a trained image pre-classification model.

[0076] Step 102: Determine one or more target objects and their object types included in the second image;

[0077] When the second image contains multiple target objects, each target object can be extracted individually, or multiple target objects can be extracted simultaneously.

[0078] In some embodiments, an image recognition model is used to identify one or more target objects and their object types in a second image.

[0079] In some embodiments, the target objects and their object types are determined by a second device; correspondingly, the method further includes: uploading a second image to a second device, so that the second device performs image detection on the second image to determine the location information and object types of one or more target objects in the second image; receiving the location information and object types of the target objects sent by the second device; and determining one or more target objects in the second image based on the location information of one or more target objects. For example, the second device can be a network device, allowing the image classification task to be implemented on the network side, thus reducing the processing burden on the mobile terminal and improving processing efficiency and performance.

[0080] Step 103: Determine the extraction model of the target object based on its object type;

[0081] In some embodiments, one object type corresponds to one extraction model for information extraction, or multiple object types correspond to one extraction model for information extraction.

[0082] In some embodiments, the method further includes: acquiring a second training sample set, the second training sample set including a second image sample and its label information, the second image sample containing a target object, and the label information including the position information of the first detection box of each element of the target object, the element type, the element identifier, and the associated element identifier; constructing an initial extraction model of the target object; and training the initial extraction model of the target object using the second training sample set to obtain a trained extraction model of the target object.

[0083] In some embodiments, the extraction model is an end-to-end neural network model.

[0084] When the target object is a mind map, the mind map extraction model includes: a backbone network, a feature pyramid network, and a detection head. The mind map is input into the backbone network to output the first feature map. The first feature map is input into the feature pyramid network for multi-scale feature extraction to output the second feature map. The first and second feature maps are input into the detection head to output the element type of each element in the mind map, the position information of the first detection box, and the position information of the second detection box.

[0085] When the target object is a flowchart, the flowchart extraction model includes an encoder and a decoder. The encoder includes a backbone network and a self-attention module. The flowchart is input into the backbone network for multi-scale feature extraction and outputs multi-scale feature maps. The first-scale feature map is input into the self-attention module and outputs a global feature map. The global feature map is fused with the feature maps of other scales to obtain multiple first feature vectors output by the encoder. The decoder outputs the type information of each element of the flowchart, the position information of the first detection box, and the position information of the second detection box from the multiple first feature vectors.

[0086] Step 104: Use the target object extraction model to extract image information of the target object, and obtain the attribute information of multiple elements of the target object and the relationship information between the elements;

[0087] The attribute information and association information are used to represent at least the position and association relationship of multiple elements that make up the target object. In some embodiments, the attribute information includes the element type of the current element and the position information of the first detection box, and the association information includes the position association information of the first detection box and the second detection box of the current element, wherein the first detection box is the bounding rectangle of the current element and the second detection box is the bounding rectangle of the associated elements of the current element.

[0088] For example, the position information of the first detection box includes the coordinates of the upper left corner and the lower right corner of the first detection box, and the position association information includes the coordinates of the upper left corner of the first detection box and the coordinates of the upper left corner of the second detection box.

[0089] For example, the position information of the first detection frame includes the first center coordinates and size; the position association information includes the first center coordinates of the first detection frame and the second center coordinates of the second detection frame. For example, the second center coordinates can specifically be the relative coordinates of the center of the second detection frame with respect to the center of the first detection frame.

[0090] A bounding box (bbox) is a rectangular box used to describe the position and extent of a target object in an image. It is typically determined by four key parameters: x, y, width, and height. x and y represent the coordinates of the top-left corner (or center point) of the rectangle in the image coordinate system, while the width and height determine the size of the rectangle. Using the bbox, we can accurately determine the specific position and size of the target object in the image.

[0091] It should be noted that attribute information and association information can be represented separately or combined. For example, the center coordinates of the first detection box are (x1, y1), and its size is (w, h); the center coordinates of the second detection box are (x2, y2). Attribute information can be represented by a quadruple [x1, y1, w, h], and association information can be represented by a quadruple [x1, y1, x2, y2]. Alternatively, attribute information and association information can be represented by a hexadecimal [x1, y1, w, h, x2, y2].

[0092] The target objects include at least one of the following types of charts: mind map, flowchart, pie chart, bar chart;

[0093] Mind maps contain at least one of the following element types: root node, intermediate node, leaf node, and thick node.

[0094] The element types of a flowchart include at least one of the following: process, decision, subprocess, start / end, document, data, condition, flow direction;

[0095] The element types of a pie chart include at least one of the following: icon title, legend, center point, sector point, and text area;

[0096] The element types of a bar chart include at least one of the following: chart title, bar, x-axis name, x-axis value, y-axis name, y-axis value, and text area.

[0097] In some embodiments, attribute information includes at least element text information;

[0098] The extraction model is configured to perform the following steps: detect multiple elements of the target object and obtain the first coordinates and size of the detection boxes of the multiple elements as attribute information of the multiple elements; determine the number of detection boxes of the target object; if the number of detection boxes is greater than or equal to a first threshold, extract text information based on the first coordinates and size of the third detection box to obtain the text information of the third detection box, wherein the third detection box is part or all of the detection boxes in the target object; if the number of detection boxes is less than the first threshold, extract text information from the target object to obtain the text information of the target object; and determine the text information of the fourth detection box based on the text information of the target object and the first coordinates and size of the fourth detection box.

[0099] When the number of bounding boxes is small (less than 50), optical character recognition (OCR) can be performed directly on the entire image. Then, using the mind map node detection boxes as a reference, the OCR text recognition content is assembled to obtain the text information of the corresponding node. When the number of bounding boxes is large (greater than or equal to 50), text recognition is performed on each node separately to ensure the accuracy of text recognition.

[0100] In some embodiments, the attribute information includes at least: element identifier, element type, style, padding, and border.

[0101] Step 105: Convert the attribute information of multiple elements of the target object and the association information between the elements into the target file format to obtain the target file of the target object.

[0102] The target file format can be a unified and standardized format that can solve the interoperability problems faced by various applications, fields, systems, cross-platforms, and ecosystems based on universality, readability, progressiveness, modularity, scalability, and unboundedness.

[0103] The target file format can be a custom file format or a standard file format. For example, standard file formats can be XMind, JSON, or XML.

[0104] In some embodiments, the method further includes: acquiring a first image to be processed.

[0105] For example, the user interface of the first application displays the content of the first file; a first image is obtained from the first file. The first application is used to display the content of the first file, and the first application can also be used to edit the content of the first file. By extracting image information from the target object in the image format of the first file, an editable extraction result is obtained, and then the extraction result can be edited again.

[0106] The first file can be in text format, such as docx, pdf, or PPT. It can also be in image format, such as jpg, jpeg, or gif.

[0107] More specifically, obtaining a first image from a first file includes: in response to a first image selection operation, obtaining a first image from the currently displayed content of the first file; or, determining one or more extractable images from the first file; using one or more extractable images as the first image to be processed; or, determining one or more extractable images from the first file; and in response to a second image selection operation, obtaining the first image from one or more extractable images.

[0108] In other words, a first image can be obtained from the currently displayed content of a first file based on a first image selection operation. For example, the first image selection operation can be a user click, a screenshot, etc., or it can be obtained from all extractable images in the first file based on a second image selection operation. Alternatively, all extractable images in the first file can be used as the first image.

[0109] The extractable images in the first file can specifically be all images included in the first file, or images that meet preset conditions. For example, the method further includes: determining a first object type to be processed; and determining one or more extractable images from the first file based on the first object type. The first object type is an extractable object type, and images containing the first object type are determined to be extractable images.

[0110] In some embodiments, obtaining a first image from a first file includes: activating a camera in response to a third image selection operation; and obtaining the first image captured by the camera.

[0111] In other words, a first image can also be acquired through a camera, and the third image selection operation can be a camera activation operation. For example, a third application can activate the camera to capture a target object, acquire a first image, and then extract information about the target object based on the image information extraction method provided in this application embodiment. Specifically, the third application can be an application that includes a "scan" function, which is activated to capture a target object.

[0112] In some embodiments, the method further includes: displaying one or more extractable object types in response to an image information extraction operation; and performing the steps of the image information extraction method in response to a selection operation of a first object type, wherein the object type of the target object belongs to the first object type. Thus, by displaying one or more extractable object types to remind the user, the user is prevented from attempting to extract unsupported object types, which would negatively impact the user experience.

[0113] The first object type is an extractable object type, which includes the object type of the target object. In some embodiments, the object type of the target object is a subtype of the first object type. For example, when the first object type is a chart, the chart can be further subdivided into subtypes such as mind map, flowchart, pie chart, bar chart, line chart, and scatter plot. The first object type is a view, which can be further subdivided into computer view or engineering view.

[0114] In some embodiments, the object type of the target object is a first object type, for example, the first object type is a chart and the first object type is a formula.

[0115] In some embodiments, the method further includes: displaying the extraction results of the target object in a first display area based on the target file of the target object. Showing the extraction results to the user facilitates related processing of the extraction results, such as editing, saving, inserting, copying, and sharing.

[0116] In some embodiments, the method further includes: displaying the target object in a second display area, wherein the first display area and the second display area are different display areas of the same user interface; and comparing the extraction result of the target object with the target object (which can also be understood as the original image of the target object) to visually demonstrate the extraction effect to the user.

[0117] In some embodiments, the method further includes: in response to an insertion operation of the extraction result of the target object, inserting the target file of the target object into a first file to which the target object belongs, to obtain a second file; and displaying the target object and its extraction result in the user interface of a first application based on the second file; or,

[0118] The method further includes: in response to a replacement operation of the extraction result of the target object, replacing the target object in the first file with the target file of the target object to obtain a third file; and displaying the extraction result of the target object in the user interface of the first application based on the third file.

[0119] In some embodiments, when the first image includes multiple target objects, the method further includes: determining the selected first target object in response to a target object selection operation; and obtaining an insertion or replacement operation of the extraction result of the first target object.

[0120] In some embodiments, the method further includes: editing the extraction results of the target object to obtain a new target file.

[0121] For example, the extraction results are edited in the first application. The first application has editing functions. After obtaining the extraction results of the target object, the extraction results are added to a first file, and then the extraction results are displayed through the first application, and the extraction results are edited using the editing tools of the first application.

[0122] In some embodiments, editing the extraction result of the target object to obtain a new target file includes: displaying a first editing menu in response to an editing operation of the target object, the first editing menu including one or more editing objects; displaying a second editing menu of the first editing object in response to a selection operation of the first editing object, the second editing menu including one or more editable elements of the first editing object; and obtaining editing information based on the second editing menu to edit the extraction result of the target object to obtain a new target file.

[0123] When the target object is a chart, one or more editable objects include at least one of the following: data source, chart type, and chart element; when the first editable object is a data source, the second edit menu includes a data table, data table insertion operations, and deletion operations; when the first editable object is a chart type, the second edit menu includes the original chart type and one or more alternative chart types; when the first editable object is a chart element, the second edit menu includes the target object's original style and element settings.

[0124] In some embodiments, when the target object is a pie chart or a bar chart, the element settings include at least one of the following: element display control, element position setting, and element content setting; when the target object is a mind map, the element settings include at least one of the following: add branch, delete branch, insert branch element, delete branch element, element style setting, and element content setting; when the target object is a flowchart, the element settings include at least one of the following: insert element, delete element, convert element type, set element content, and set element connection relationship.

[0125] Element style settings can include at least one of the following: color, font, shape, etc.

[0126] In some embodiments, the method further includes: in response to a first format conversion operation, converting the target file into a first file format to obtain a fourth file; the fourth file is used to display the target object in the user interface of a second application, the first file format being a file format supported by the second application; and in response to a second format conversion operation, converting a second file or a third file into a second file format to obtain a fifth file.

[0127] The first format conversion operation can be understood as a "save as" operation on the extracted result, while the second format conversion operation can be understood as a "save as" operation on a second or third file. The "save as" operation facilitates file transfer and adapts to different applications.

[0128] The first file format can be an image format such as jpg, jpeg, or gif, and the second file format can be a standard file format, such as a text format like docx, pdf, or PPT, or an image format such as jpg, jpeg, or gif.

[0129] To better illustrate the purpose of this application, the target audience will use diagrams as examples to further illustrate the embodiments of this application.

[0130] Figures 2 and 3 are schematic diagrams of the single-image information extraction interface in this embodiment of the application. After opening the file using an application with image information extraction capabilities, the user can select the image by long-pressing the image area in the document, or by taking a screenshot. After selecting the image, the user can use the "AI Document Assistant" in the lower left corner of Figure 2 (left image) to invoke the relevant functions, corresponding to the image information extraction operation. The "AI Document Assistant" menu in Figure 2 (right image) displays one or more object types that can be extracted, specifically including "Extract Charts" and "Extract Formulas". The user can select the "Extract Charts" function to extract information from a single chart.

[0131] When the user selects "Extract Chart", the AI ​​system reads the image selected by the user and then runs the image information extraction algorithm provided in this application embodiment to parse and reconstruct the image content. The waiting interface during AI system processing is shown in the left image of Figure 3, which prompts "Generating chart".

[0132] After the AI ​​system completes its image information extraction algorithm, it will display the extracted chart in the lower area of ​​the screen, as shown in the right image of Figure 3. Users can further use the "Replace Original Image" button to replace the original image with the extracted chart in the original document, facilitating secondary editing of the chart. Users can also use the "Insert Below" button to insert the extracted chart below the corresponding chart in the original document for easy comparison of the extraction results.

[0133] Figures 4 and 5 are schematic diagrams of the multi-image information extraction interface in this embodiment of the application. When the image selected by the user contains multiple chart areas, the user can still use the "Extract Charts" function through the "AI Document Assistant" page. At this time, the AI ​​system will automatically identify and segment the multiple chart areas in the original image, and extract information from the charts in each area separately.

[0134] After the AI ​​system completes its information extraction algorithm, it will present the reconstructed results in a paginated format, with each page displaying one chart or graph showing the extracted information, as shown in Figure 5. Users need to select the extracted results on each page before they can use the "Replace Original Image" or "Insert Below" function buttons.

[0135] Figure 6 is a schematic diagram of the automatic chart information extraction interface in an embodiment of this application. As shown in the left figure of Figure 6, users can also use the chart information extraction function without manually selecting charts. In this case, after the user opens the file with the application, they can directly click "AI Document Assistant" and select the type of object to be extracted in the pop-up menu bar, such as selecting the "Extract Charts" function, as shown in the right figure of Figure 6. When the user selects "Extract Charts", the system will automatically recognize all charts in image format in the original document and display an "Extract" button below each chart. The user can then further select the charts for which they wish to extract chart information, as shown in Figure 7. It should be noted that only one "Extract" button can be displayed, and clicking the "Extract" button can extract all charts in batches.

[0136] After successfully extracting the chart information, as shown in the left image of Figure 8, users can use the "Copy" function to convert the chart into an image format and copy it to other image editing tools that support image formats. As shown in the right image of Figure 8, users can also use the "Share" function to convert the extracted chart into a specific format: image, Word, PDF, or PPT.

[0137] The client-side editor supports editing pie charts, bar charts, and mind maps. For pie charts and bar charts, since their underlying data format is tabular, editing them is actually transformed into editing the underlying tabular data, as shown in Figures 9 and 10.

[0138] Editing features for pie charts and bar charts: Data sources can be freely deleted, rows and columns can be added, and content can be modified; the display position, title content, and whether to display the title can be modified; the display position and whether to display the legend can be adjusted; pie charts and bar charts can be converted to each other.

[0139] Figure 11 shows a schematic diagram of mind map editing.

[0140] Mind map editing features include: adding sibling topics and subtopics; modifying text content within topics at all levels; deleting topics at all levels; and modifying topic styles.

[0141] Figure 12 is a schematic diagram of the image information extraction interface implemented in the "Scan" function according to an embodiment of this application. A function entry for extracting chart information can also be added to the "Scan" function. When a user opens the "Scan" function, if the AI ​​algorithm recognizes that the current interface is a document and contains image categories supported by image and text information extraction, the "Scan" function can call the chart information extraction capability to extract information and present the results to the user, along with a result export function.

[0142] To better illustrate the purpose of this application, further examples are provided based on the above embodiments. As shown in Figure 13, the technical solution provided by the embodiments of this application can be divided into the following core functional modules according to the data processing flow:

[0143] Step 1301: Image pre-classification module. The edge-side pre-classification algorithm is used on the images in the user's current document (or images actively selected by the user) to determine whether the current image contains object types that the system can support for reconstruction.

[0144] Step 1302: Image Quality Enhancement Module. For supported object types, image enhancement algorithms are applied to enhance the image quality based on its actual condition. Feasible enhancement strategies include: trapezoidal correction, surface correction, moiré removal, and deblurring.

[0145] Step 1303: Information Extraction Module. Based on the chart type, corresponding extraction algorithms can be used to extract attribute information of multiple elements and the relationship information between elements for different object types, and then converted into a standardized file format to obtain the target file.

[0146] Step 1304: Rendering Module. Renders the extracted object information on the display interface based on the target file.

[0147] Specifically, in this application embodiment, the extractable object types are illustrated by flowcharts, mind maps, bar charts, and pie charts, which further illustrate the data processing flow of each module of a model.

[0148] 1. Image pre-classification module

[0149] The image pre-classification module is used to determine whether the currently input image is a type of image that the system can support for reconstruction (flowchart, mind map, bar chart, pie chart). The main technical challenge faced by the edge-side pre-classification module is how to accurately identify rejected samples in open scenes. In this embodiment, all images that do not belong to one of the following categories—flowchart, mind map, bar chart, or pie chart—are considered rejected samples. As shown in Figure 14, the three images at the top are extractable samples, and the three images at the bottom are rejected samples.

[0150] Overall, this application provides a pre-classification module that can distinguish between different types of images and reject images outside the specified size range. It can be used in common office scenarios and can also be adapted to other similar scenarios. The general operation involves taking a screenshot or photograph of a chart image in format A, opened by user software or displayed on a screen. The model then outputs the corresponding category, which is subsequently assigned to a parsing algorithm to obtain editable structured data. This greatly improves the accuracy and rationality of the assignment, as well as the user experience.

[0151] Figure 15 is a schematic diagram of the training process of the image pre-classification module in this embodiment of the application. The specific steps of the image pre-classification module are as follows:

[0152] S1: Obtain the training sample set images.

[0153] Image data collected from public sources or captured by mobile phones, including image files of mind maps, bar charts, pie charts, and flowcharts.

[0154] S2: Use the random data augmentation strategy (RandAugment) to obtain two different views.

[0155] A set of image augmentation operations is randomly selected from a predefined pool of operations, with each selected operation applied at a different intensity. For each training sample, the corresponding augmentation operation is applied in a random order according to the selected augmentation strategy. We apply the random data augmentation strategy twice to each training sample, resulting in tensor representations of two different augmented image samples.

[0156] A tensor is a multidimensional array that can be represented as a vector, matrix, or higher-dimensional data structure. The tensor representation of an image can be viewed as a two-dimensional array (or matrix), where each element represents the pixel value at a corresponding location in the image. The tensor representation of an image typically consists of a two-dimensional array (or matrix) with multiple channels. The tensor representation of an image can be understood as image data.

[0157] The main idea of ​​RandAugment is to randomly select transformations and adjust their sizes to increase the diversity of the data, thereby improving the generalization ability of the model.

[0158] S3: Use the Resizemix method to construct more blended views and obtain tensor representations (aug_images) of the blended image samples, i.e., the blended image samples.

[0159] This application employs a size adjustment mixing scheme to further expand the training data in order to simulate samples of unknown categories. Resizemix is ​​a data mixing method that combines saliency detection. Compared with other data mixing methods such as mixup, it has stronger interpretability and a more reasonable label allocation strategy. At the same time, it can also generate some mixed views more reasonably as a supplement to the training sample set.

[0160] The core idea of ​​the Resizemix method is to directly scale an image and paste it onto another image, thereby blending the data. This method avoids the problems of mislabeling and loss of object information that can occur with traditional crop-based data blending methods.

[0161] S4: Perform a deep copy operation on the tensor representation of the enhanced image samples to obtain the tensor representation of the enhanced image samples (orig_images), i.e., the enhanced image samples;

[0162] S5: Input the tensor representations of the enhanced image samples (orig_images) and the tensor representations of the mixed image samples (aug_images) into the feature extraction network (Feature Extractor) to obtain the auxiliary classification result (aux_output), the enhanced image feature embedding representation (Feature Embeddings Orig), and the mixed image feature embedding representation (Feature Embeddings Mix) output by the auxiliary classification head.

[0163] In some embodiments, the MobileNet V2 network is used as the feature extraction network, and the fully connected layers of its original classification head are replaced with linear layers with spectral norms as feature projection layers, making the gradient more stable during weight updates. Furthermore, this embodiment also uses a small multilayer perceptron (MLP) as an auxiliary classification head (which can also be understood as a classification network or classification layer) to generate auxiliary classification results aux_output. The feature projection layer receives enhanced image samples and mixed image samples as input, and after forward computation by the network, it obtains the enhanced image feature embeddings (Orig) and the mixed image feature embeddings (Mix).

[0164] A deep copy operation creates a completely new image object and copies all the data from the original image object to the new object. This way, the original image object and the new image object have independent memory addresses and data. Therefore, modifications to the original image object will not affect the new image object obtained through a deep copy.

[0165] S6: Calculate the first loss function of the auxiliary classification result and perform gradient perturbation.

[0166] For the auxiliary classification result aux_output generated in S5, the cross-entropy loss function (i.e., the first loss function) is calculated, and the gradient of the first loss function is used as the perturbation of the original features. The specific formula is as follows:

[0167] Where g represents the gradient of the cross-entropy loss with respect to the feature embedding representation; ε is a hyperparameter that controls the size of the perturbation and needs to be set manually. After obtaining the gradient perturbation, the original feature embeddings Orig are added to the perturbation to obtain the perturbed feature embeddings Att.

[0168] S7: Calculation of the second loss function.

[0169] First, the InfoNCE loss function in Figure 4 is calculated. The expression for the InfoNCE loss function is as follows:

[0170] Where q represents the current sample, k + This represents its unique positive sample. Then, the main loss function is calculated, which is the SupCon loss function in Figure 4, and its expression is as follows:

[0171] Where y represents the label of the current data, and s(·) represents the similarity calculation between labels.

[0172] S8: Fine-tuning of classification tasks based on pre-trained feature extraction networks.

[0173] After the pre-training phase, the weight parameters of the current model are fine-tuned to improve model performance. This application provides two selectable methods for fine-tuning. The first method is to freeze the feature extraction network parameters, add a new fully connected layer with SpectralNorm as the classification head, and train the classification task on the training sample set, as shown in Figure 16, replacing the original auxiliary classification head with the new classification head. The second method is to not introduce any new structure, directly load the original network weights, but only update the parameters of the auxiliary head.

[0174] 2. Image quality enhancement module

[0175] The image quality enhancement module first uses an image quality pre-analysis algorithm to determine the image enhancement method (one or more) to be applied to the input image. Once the required image quality enhancement method is determined, the selected enhancement method is applied to the image sequentially according to the priority from left to right in Figure 17, ultimately obtaining the enhanced output image.

[0176] 2.1 Image Quality Pre-analysis Algorithm

[0177] The image quality pre-analysis algorithm is used to determine whether the current input image needs specific image enhancement processing (trapezoidal correction, surface correction, moiré removal, and deblurring). The image types collected in this section include: images taken from tilted positions, images with curved surfaces, images with moiré patterns, and images with blurriness; the collected image samples are used to train the image quality pre-analysis model.

[0178] 2.2 Trapezoidal Correction

[0179] Trapezoidal correction corrects the distortion caused by tilted shooting to a normal shape, as shown in Figure 18. The complete process of trapezoidal correction is shown in Figure 19. The complete steps of trapezoidal correction are as follows:

[0180] S1: Obtain the original document image to be corrected.

[0181] S2: Document Edge Probability Prediction

[0182] S2.1: Input the document image to be corrected into the document edge detection model to obtain the document edge probability map, denoted as the first document edge. In the document edge probability map, the value of each pixel represents the probability that the corresponding pixel in the image to be corrected belongs to the document edge, and the value range is [0.0, 1.0]. The larger the value, the higher the probability that the pixel belongs to the document edge.

[0183] S2.2: Binarize the edges of the first document to obtain the edges of the second document. Use a threshold binarization method to binarize the edges of the first document to obtain the edges of the second document.

[0184] The specific process is as follows: traverse the edge probability value of each pixel. If the probability value is greater than a set threshold (e.g., 0.85), then reassign the edge probability of that pixel to 1; otherwise, assign it to 0. Equivalently, traverse the visualized edge probability map. If the value of a pixel is greater than 255, then assign the value of that pixel to 255; otherwise, assign it to 0, thus obtaining the binary map of document edges, denoted as the second document edge.

[0185] S2.3: Filter the second document edges to remove erroneous edges and obtain the third document edges. Since the second document edges may contain some "edge blocks" within the background or document area, these need to be filtered out. The filtering method is as follows: using a connected component algorithm, find all connected edges in the second document edges, calculate the area of ​​each connected edge, and if the area is less than a set threshold, filter it out, keeping only edges with an area greater than the threshold. The filtered result is recorded as the third document edges.

[0186] S2.4: The third document edge is thinned to obtain the fourth document edge. In the third document edge, the edge thickness is typically greater than one pixel. If line detection is performed directly, many invalid lines will be generated. Therefore, it is necessary to thin the edge. In this embodiment, we use the ZhangSuen thinning algorithm for edge skeleton extraction, denoted as the fourth document edge.

[0187] S3: Document boundary line fitting

[0188] S3.1: Using the progressive Hough transform algorithm, straight lines are detected on the edges of the fourth document to obtain the first set of straight lines, and the straight lines are converted from rectangular coordinate representation to the following parameterized form:

[0189] Where (x1,y1) and (x2,y2) represent two distinct points on the line.

[0190] S3.2: Sort and cluster the lines in the first set of lines, and take the cluster center of each class to form the second set of lines, thereby realizing the merging of lines in the first set of lines.

[0191] The lines in the first set of lines are merged as follows:

[0192] 1) Sort the lines in the first set of lines in ascending order of their ρ values;

[0193] 2) Define all line families as C, the first type of line family as c1, and initialize them as c1 = {l1,} and C = {c1,} respectively, where l1 is the first line in the sorted first line set;

[0194] 3) Select the next straight line li (2≤i≤n), and calculate c. -1 center c -1 This represents the last type of line family in the current C, if l i and If the difference between ρ and the difference between θ are both less than the set threshold, then l i Add to c -1 In the middle; otherwise, with l i Based on this, a new class of linear families c is created. j ={l i ,}, and c j Add it to C;

[0195] 4) Repeat step 3) until all lines have been traversed, finally obtaining C = {c1, c2, ..., c m}, each c iThere is at least one straight line in the equation; if there are multiple straight lines, then they are approximately the same straight line.

[0196] 5) After completing the above linear clustering, calculate c for each type of linear cluster. i center All The set This is the result of merging straight lines, denoted as the second set of straight lines.

[0197] S3.3: Filter out lines that are almost horizontal or vertical and close to the image boundary in the second set of lines. Then use the disjoint-set data structure algorithm to cluster the filtered set of lines. If the number of clusters is less than 4, cancel the previous filtering. The filtered set of lines is called the third set of lines. The number of lines in the third set of lines is less than or equal to the number in the second set of lines.

[0198] S4: Document vertex positioning

[0199] S4.1: Calculate the angle between any two lines in the third set of lines. If the angle satisfies the threshold range (e.g., (50°, 90°)), then calculate the intersection of these two lines. All such intersections form a candidate intersection set.

[0200] S4.2: Using the K-Means algorithm, the intersection points in the candidate intersection point set are divided into 4 types of point clusters, denoted as C1, C2, C3, and C4.

[0201] S4.3: Based on the set filtering rules, filter the four types of point clusters respectively. The centers of the four types of point clusters after filtering are the four vertices of the document.

[0202] The four types of point clusters are filtered separately. The filtering process is as follows:

[0203] 1) Based on whether the intersection point is located within the image, group the point cluster C. i The intersection points in (1≤i≤4) are divided into "intra-graph point clusters" C. i,inner and “outside-the-figure cluster” C i,outer C i =C i,inner +C i,outer ;

[0204] 2) If C i,outer Empty, i.e., point cluster C i If there are no diplomatic points in the map, then for C... i,inner Filtering is performed, and the filtering method is as follows: If C i,inner If there is only one intersection point, then keep it directly; if C i,inner If there are multiple points, only the points whose coordinates lie on the edge of the third document will be retained; if C i,innerIf multiple points in the image are not located on the edge of the third document, then only the point closest to the center of the image is retained.

[0205] 3) If C i,outer If there is an intersection point (corresponding to the "missing document corner" problem caused by incomplete shooting), then clear C. i,inner and C i,outer Filtering is performed, and the filtering method is as follows: If C i,outer If there is only one intersection point, then keep it directly; if C i,outer If there are multiple intersection points, then only the intersection point closest to the center of the image is retained.

[0206] 4) After obtaining the four types of point clusters after filtering, calculate the center of each type of point cluster. The coordinates of the four centers are the coordinates of the four vertices of the document, denoted as P = {(x...} i ,y i ),1≤i≤4}, and the document vertex positioning is now complete.

[0207] S5: Perspective Transformation

[0208] The perspective transformation process is as follows:

[0209] 1) Determine if the quadrilateral formed by the four document vertices P is reasonable. The criteria are: whether the quadrilateral occupies more than 10% of the area of ​​the entire image, and whether the four interior angles of the quadrilateral are within the range of [50°, 130°]. If the quadrilateral meets the above two conditions, continue with the subsequent perspective transformation; otherwise, skip the subsequent steps and return to the original image directly.

[0210] 2) Based on the coordinates of the four document vertices, determine whether to perform a padding operation. If a padding operation is performed, calculate the width of the padding to be filled in the four directions of top, bottom, left, and right of the image to be corrected, and use the padding width to fill the original image, and update the coordinates of the four vertices accordingly.

[0211] 3) Based on the coordinates of the four vertices, calculate the four target vertices of the document in the target plane;

[0212] 4) Calculate the perspective transformation matrix M based on the 4 vertices and 4 target vertices, and then perform perspective transformation on the padded image to be corrected based on M to complete the document correction.

[0213] 2.2 Image Deblurring

[0214] Image deblurring can sharpen blurry images caused by long-distance shooting, enabling subsequent chart information extraction algorithms to extract chart information more accurately. The detailed steps of image deblurring are as follows.

[0215] S1: Acquire clear image data

[0216] Collect a dataset of 5000 clear images, referred to as Dataset 1.

[0217] S2: Add random noise

[0218] Random noise is added to dataset 1. Each image in dataset 1 is processed sequentially, and one of four strategies—Gaussian noise, salt-and-pepper noise, Poisson noise, or no noise—is selected with a probability distribution [0.3, 0.3, 0.3, 0.1]. In this embodiment, the random selection range for the Gaussian noise level parameter is [0, 32], the random selection range for the Poisson noise level parameter is [0, 32], and the random selection range for the salt-and-pepper noise level parameter is [0, 0.04], resulting in dataset 2.

[0219] S3: Add motion blur

[0220] Dataset 2 was simulated by adding motion blur of varying degrees and angles to simulate lens shake during real-world mobile phone photography. In this embodiment, the proportion of images with added motion blur was 20%, and the adjustable motion blur angle and degree parameters ranged from [10, 20] and [1, 180], respectively. Dataset 3 was then obtained.

[0221] S4: Add tadpole-shaped deformation

[0222] For dataset 3, the tadpole-like deformation effect of QR codes at different distances in a real environment was simulated by selecting methods based on average pooling and Gaussian blurring. This embodiment uses three methods: multi-degree tadpole-like degradation, multi-degree Gaussian blurring degradation, and no degradation, in a ratio of 0.7:0.2:0.1. Tadpole-like degradation and Gaussian blurring degradation each use two degradation levels, mild and severe, in a ratio of 0.8:0.2. During the degradation process, for diverse QR code images, this degradation step calculates the size of the average pooling kernel based on the size of the QR code's positioning code; different density QR codes use different pooling kernel sizes. In this embodiment, the pooling window size and Gaussian blur kernel size are defined as 1 / 5 and 1 / 4 of the positioning code size in the mild and severe levels, respectively, and the pooling step size is fixed at 1 / 4 of the pooling window size. This yields dataset 4.

[0223] S5: Training the deblurring model

[0224] For example, the deblurring model can be the deep learning model DeblurNet. Using datasets 1 and 4, the deep learning model DeblurNet is trained to achieve image deblurring. DeblurNet adopts an encoder-decoder structure, mainly consisting of three parts: the contracting path (CP), the expanding path (EP), and skip connections, which are symmetrically distributed. The contracting path process uses a max-pooling layer (ML) + convolution to perform four consecutive downsampling operations. As the spatial resolution decreases, the number of channels increases, which can fully capture the contextual information in the image. The expanding path process uses bilinear interpolation (BI) + convolution to perform four consecutive upsampling operations. As the number of channels decreases, the spatial resolution increases, which can accurately locate the information of interest in the image. Furthermore, the skip connection process is implemented through concatenation, which can retain more dimensional or positional information, allowing the deep network layers for image reconstruction to freely choose between shallow and deep features. This is particularly suitable for recovering local and global information in blurred images.

[0225] 3. Information Extraction Module

[0226] As shown in Figure 20, for the input image, the information extraction module first uses a layout analysis algorithm to identify chart regions from the image and determine the chart type for each region. Then, based on the chart type, the chart is sent to the corresponding information extraction module. The following section focuses on the end-to-end solution for extracting information from four types of charts.

[0227] 3.1: Mind Map Information Extraction Algorithm

[0228] Mind map information extraction algorithms extract structured data from mind maps (such as JPG, PNG, and BMP formats) and convert this data into mind map files in specific formats (such as XMind and JSON formats).

[0229] A flowchart information extraction algorithm is implemented using a mind map extraction model. The training method for the mind map extraction model is as follows:

[0230] S1: Obtain a mind map image sample.

[0231] Collect and annotate mind map images. As shown in Figure 21, the types of image annotations include: root node, intermediate node, leaf node, and cluster node.

[0232] Each rectangular bounding box has the following attributes: label and point (boundary coordinates), where point represents the coordinates of the top-left and bottom-right corners of the rectangle. In addition, each node is labeled with its ID and the ID of its parent node. A complete label for a node can be represented as:

[0233] {"parent_ID":1,"label":"node","node_ID":2,"points":[[1482,364],[1694,462]]}.

[0234] S2: Build the model. Use the image samples obtained in S1 to train the model. Use the model to obtain the position and size of the detection box (i.e., the first detection box) of each node, and the position of the detection box of its parent node (i.e., the second detection box).

[0235] This application presents a novel end-to-end mind map reconstruction model. Based on object detection, this model combines node location identification and inter-node relationship identification, outputting the target node's location and the coordinates of its parent node's center point in a single output within the object detection header. The model design primarily involves the following optimizations and modifications:

[0236] 1) In the regression phase of object detection bounding boxes, two new regression targets were added to represent the center coordinates of the parent node.

[0237] 2) Use the relative position of the parent node for regression. Since the relative value of the target relative value is small because the distance between the parent node and the current node is very close, using the relative position makes it easier to train.

[0238] 3) The Generalized Focal Loss (GFL) was used for weighted calculation.

[0239] 4) Assign higher loss weights to samples whose parent nodes cannot be accurately predicted. This is for difficult sample mining.

[0240] Based on the optimized model, the mind map image input by the user is fed into the mind map structure reconstruction model, which can simultaneously obtain multiple detection boxes and parent node positions, thus obtaining the key structural information of the mind map.

[0241] S3: Perform optical character recognition on mind maps.

[0242] The coordinates of the detection box {x1,y1,x2,y2} of each node in the mind map were obtained through S2, and the text information was extracted from them.

[0243] When the number of bounding boxes is small (less than 50), OCR text recognition is performed directly on the entire image. Then, using the mind map node detection boxes as a reference, the content of the OCR text recognition is assembled to obtain the text information of the corresponding node. When the number of bounding boxes is large (greater than or equal to 50), text recognition is performed on each node separately to ensure the accuracy of text recognition.

[0244] S4: Generate mind map structured data

[0245] The results obtained from S2 and S3 are assembled to obtain a standardized file format.

[0246] 3.2: Flowchart Information Extraction Algorithm

[0247] The core of flowchart information extraction algorithms is to extract structured data from flowcharts (such as JPG, PNG, and BMP formats) and convert this data into flowchart files in a specific format (such as XMind and JSON formats).

[0248] A flowchart extraction model is used to implement the flowchart information extraction algorithm. The training method of the flowchart extraction model is as follows:

[0249] S1: Obtain the flowchart image.

[0250] Collect and annotate flowchart images. As shown in Figure 22, the types of image annotations include: process, decision, sub-process, start / end, document, data, condition, flow direction, etc.

[0251] Rectangular label boxes all have the following attributes: label and coordinates point, where point represents the coordinates of the top left and bottom right corners of the rectangle.

[0252] In addition, each node is labeled with its ID and the ID of its parent node. A complete label for a node can be represented as:

[0253] {"parent_ID":[1,13,14],"label":"'Process'","node_ID":2,point[(38,266),(198,385)]}.

[0254] S2: Build a model and use the model to obtain the bounding box and relationship recognition results for each node.

[0255] This application's embodiments model the extraction of relationships in a flowchart as the prediction of relationship pairs, where a relationship pair is represented as (coordinates of the parent node, coordinates of the child node). The model employs an encoder-decoder architecture, containing two decoders for extracting node relationships and detecting target nodes in the flowchart, respectively.

[0256] In the encoder, ResNet serves as the backbone network, outputting feature maps at three different scales. The smallest feature map is input into the self-attention module for computation, capturing features at the full-image level. Subsequently, it is fused with feature maps from the other two scales to obtain the final output of the encoder.

[0257] On the decoder side, a deformable attention mechanism is employed to reduce computational complexity while maintaining the ability to model long distances. After obtaining multiple feature vectors from the encoder output, the model needs to select 300 vectors as query vectors for the decoder. The selection criteria are as follows: multiple feature vectors from the encoder output are fed into the feature evaluation module, which can parse the information in the feature vectors to obtain the estimated relation pairs and their corresponding confidence scores. Then, based on the confidence scores, the model selects from the feature vectors output by the encoder. Based on the selected feature vectors, the decoder outputs the position and relation information of the target text boxes.

[0258] S3: Perform optical character recognition on the flowchart.

[0259] The coordinates of the detection box {x1,y1,x2,y2} of each node in the flowchart were obtained through S2, and the text information was extracted from them.

[0260] When the number of bounding boxes is small (less than 50), OCR text recognition is performed directly on the entire image. Then, using the mind map node detection boxes as a reference, the content of the OCR text recognition is assembled to obtain the text information of the corresponding node. When the number of bounding boxes is large (greater than or equal to 50), text recognition is performed on each node separately to ensure the accuracy of text recognition.

[0261] S4: Generate flowchart structured data

[0262] The results obtained from S2 and S3 are assembled to obtain a standardized file format.

[0263] 3.3: Algorithm for extracting information from bar charts and pie charts

[0264] Bar charts and pie charts use the same technical approach, employing an end-to-end neural network to extract the relationships between key elements in the chart.

[0265] An extraction model is used to implement algorithms for extracting information from bar charts and pie charts. The training method for the extraction model is as follows:

[0266] S1: Collect dataset

[0267] Collect and label pie charts and bar charts. For pie charts, the labeling information includes: chart title, legend, sectors, and other explanatory text. For bar charts, the labeling information includes: chart title, column type, x-axis name, x-axis value, y-axis name, and y-axis value.

[0268] S2: Object Representation Method

[0269] As shown in Figure 23, for a pie chart, the objects to be identified include: chart title, legend, sectors, and other explanatory text. The title, legend, and other explanatory text are represented by a rectangle, with four values ​​[x, y, w, h], where x and y represent the coordinates of the center point of the rectangle; w represents the width of the rectangle; and h represents the height of the rectangle. For sectors, an indirect representation is used. A sector is represented by its midpoint (represented by the center detection box in Figure 23) and the intersection points of each sector on the arcs (represented by the detection boxes on the arcs in Figure 23). These points are enlarged into a 5x5 rectangle and then represented using the rectangle representation method. In this representation method, a sector can be represented by the midpoint of the sector and the intersection points of two arcs.

[0270] For bar charts, the objects that need to be identified include: chart title, bar, x-axis name, x-axis value, y-axis name, y-axis value, and other text areas. All of these objects are represented by rectangles [x, y, w, h].

[0271] S3: Relational Representation Methods

[0272] In a pie chart, the relationships between objects include:

[0273] 1) The relationship between the legend and each sector;

[0274] 2) The relationship between the extra text box and each sector.

[0275] In a bar chart, the relationships between objects include:

[0276] 1) The relationship between each column and the legend;

[0277] 2) The relationship between each bar and the text box.

[0278] For any relation, this application embodiment uses a quadruple to represent it: [x1, y1, x2, y2]. Where x1, y1 represent the coordinates of the center point of the column or sector in the current relation; x2, y2 represent the coordinates of the center point of another object in the current relation.

[0279] In terms of representation, each relation and each key element is represented by four values.

[0280] S4: Constructing the extraction model

[0281] The extraction model employs an encoder-decoder architecture. It comprises two decoders: one for node recognition (key features) and the other for node relationship recognition. The encoder extracts image features and outputs relationship pairs and confidence scores. The decoders filter and optimize the encoder's output to obtain the final full-image node relationships. Simultaneously, the image features extracted by the encoder are passed through the node recognition decoder to output the node recognition results.

[0282] For the encoder, a CNN-like model, such as ResNet, is used as the backbone network, which can output feature maps at multiple scales. After the backbone network has finished computing, the smallest feature map is input into the self-attention module for computation to capture features at the full image level. Then, feature maps at the other two scales are fused to obtain the final output of the encoder.

[0283] After obtaining multiple feature vectors from the encoder output, the model needs to select 300 vectors (maximum number of detection nodes and relation extractions) as queries for the decoder. The selection criteria are as follows: multiple feature vectors from the encoder output are fed into the feature evaluation module, which can parse the information in the feature vectors to obtain the estimated relation pairs and their corresponding confidence scores. Then, based on the confidence scores, the model selects from the feature vectors output by the encoder.

[0284] On the decoder side, a deformable attention mechanism is employed to reduce computational complexity while maintaining the ability to model long distances. The deformable Transformer performs excellently in computer vision tasks, particularly object detection. By introducing a deformable attention mechanism, it effectively handles spatial variations in images, improving the model's understanding of image content. In the standard Transformer model, the attention mechanism considers the interactions between all elements in the input sequence, leading to significant computational costs when processing images, as images are typically converted into a large number of pixels or feature point sequences. The deformable Transformer, through its deformable attention mechanism, allows the model to focus on only a subset of key regions of the image, rather than the entire image. This approach not only reduces computation but also improves the efficiency and effectiveness of image processing. Deformable attention allows the model to dynamically select small subsets of key sampling locations for attention computation within each attention head; these locations are dynamically determined based on the input data. This allows the model to focus more on important information in the image, improving its performance and generalization ability.

[0285] S5: Extracting Chart Information Using Models

[0286] In practical applications, for input pie charts or bar charts, the model obtained by S4 can directly obtain the attribute information of all nodes in the chart and the relationship information between nodes.

[0287] The image information extraction method provided in this application has the following beneficial effects:

[0288] Users can use this feature to quickly and easily extract relevant information from image-based charts and export them to third-party editors (such as Word and Excel) for editing, which can greatly improve the efficiency of users editing image-based charts on mobile devices.

[0289] Regarding chart information extraction, this application's embodiments design and implement end-to-end deep neural network models for four types of charts. The end-to-end approach makes the model more robust in complex backgrounds and noisy real-world environments. Furthermore, in actual model and product performance, the end-to-end approach significantly outperforms other existing solutions. This function supports intelligent recognition and information extraction from flowcharts, mind maps, bar charts, and pie charts in documents, and allows for sharing or copying to third-party editing tools for secondary editing.

[0290] In terms of data flow, standardized file formats enable a complete data chain from collection to use. From collection to transfer and sharing, standardized file formats provide a low-cost, easy-to-transmit, and easy-to-understand format, establishing a complete data chain on the device. The stored data has strong recognizability, and the entire device will provide content support to third-party applications and service layers in a complete data format, shortening the path for content to reach users and reducing the usage costs of complex functions and scenarios.

[0291] To implement the method of the embodiments of this application, based on the same inventive concept, the embodiments of this application also provide an image information extraction device, as shown in FIG24. The image information extraction device 240 includes:

[0292] Image enhancement unit 2401 is configured to enhance the first image when the first image to be processed includes a target object, thereby obtaining an enhanced second image;

[0293] The determining unit 2402 is configured to determine one or more target objects and their object types included in the second image; and to determine the extraction model of the target objects based on their object types.

[0294] Extraction unit 2403 is configured to extract image information of target object using the target object extraction model, and obtain attribute information of multiple elements of target object and the association information between elements;

[0295] Extraction unit 2403 is also configured to convert the attribute information of multiple elements of the target object and the association information between the elements into the target file format to obtain the target file of the target object.

[0296] In some embodiments, the image information extraction device 240 further includes: a display unit configured to display the contents of a first file in the user interface of a first application; and a processing unit configured to acquire a first image from the first file.

[0297] In some embodiments, the processing unit is configured to, in response to a first image selection operation, obtain a first image from the currently displayed content of a first file; or, determine one or more extractable images from the first file and use one or more extractable images as the first image to be processed; or, determine one or more extractable images from the first file and, in response to a second image selection operation, obtain the first image from one or more extractable images.

[0298] In some embodiments, the processing unit is further configured to determine a first object type to be processed; and based on the first object type, to determine one or more extractable images from a first file.

[0299] In some embodiments, the processing unit is further configured to activate the camera in response to a third image selection operation and acquire a first image captured by the camera.

[0300] In some embodiments, the display unit is further configured to display one or more extractable object types in response to an image information extraction operation; and to trigger the execution of an image information extraction method step in response to a selection operation of a first object type, wherein the object type of the target object belongs to the first object type.

[0301] In some embodiments, the display unit is further configured to display the extraction result of the target object in a first display area based on the target file of the target object.

[0302] In some embodiments, the display unit is further configured to display the target object in a second display area, wherein the first display area and the second display area are different display areas of the same user interface.

[0303] The processing unit is further configured to, in response to an insertion operation of the extraction result of the target object, insert the target file of the target object into the first file to which the target object belongs, to obtain a second file; the display unit is further configured to, based on the second file, display the target object and its extraction result in the user interface of the first application; or,

[0304] The processing unit is further configured to replace the target object in the first file with the target file of the target object in response to the replacement operation of the extraction result of the target object, thereby obtaining a third file; the display unit is further configured to display the extraction result of the target object in the user interface of the first application based on the third file.

[0305] In some embodiments, when the first image includes multiple target objects, the processing unit is further configured to, in response to a target object selection operation, determine the selected first target object; and obtain an insertion operation or a replacement operation of the extraction result of the first target object.

[0306] In some embodiments, the processing unit is configured to edit the extraction results of the target object to obtain a new target file.

[0307] In some embodiments, the processing unit is configured to, in response to an editing operation on a target object, display a first editing menu, the first editing menu including one or more editing objects; in response to a selection operation on the first editing object, display a second editing menu on the first editing object, the second editing menu including one or more editable elements of the first editing object; and, based on the second editing menu, obtain editing information to edit the extraction result of the target object to obtain a new target file.

[0308] In some embodiments, when the target object is a chart, one or more editable objects include at least one of the following: data source, chart type, chart element;

[0309] When the first editing object is a data source, the second editing menu includes data table, data table insertion operation, and data table deletion operation;

[0310] When the first editing object is a chart type, the second editing menu includes the original chart type and one or more alternative chart types;

[0311] When the first editing object is a chart element, the second editing menu includes the target object's original style and element settings.

[0312] In some embodiments, when the target object is a pie chart or a bar chart, the element settings include at least one of the following: element display control, element position setting, and element content setting;

[0313] When the target object is a mind map, the element settings include at least one of the following: add branch, delete branch, insert branch element, delete branch element, element style settings, and element content settings;

[0314] When the target object is a flowchart, the element settings include at least one of the following: element insertion, element deletion, element type conversion, element content settings, and element connection relationship settings.

[0315] In some embodiments, the processing unit is configured to, in response to a first format conversion operation, convert a target file into a first file format to obtain a fourth file; the fourth file is configured to display the target object in the user interface of a second application, wherein the first file format is a file format supported by the second application; and in response to a second format conversion operation, convert a second file or a third file into a second file format to obtain a fifth file.

[0316] In some embodiments, the processing unit is further configured to determine that the first image is an unextractable image if the first image does not contain the target object, and skip the steps of the image information extraction method.

[0317] In some embodiments, the processing unit is further configured to output a first prompt message if the first image does not include the target object, so as to prompt the user that the first image is an unextractable image.

[0318] In some embodiments, the image information extraction device 240 further includes: an image pre-classification unit, configured to pre-classify a first image using an image pre-classification model to obtain a classification result, the classification result being configured to indicate whether the first image includes a target object, the image pre-classification model including a feature extraction network and a classification head, the feature extraction network being configured to extract features from the first image to obtain a feature embedding representation, and the classification head being configured to perform image classification based on the feature embedding representation to determine whether the first image is a target object.

[0319] In some embodiments, the image pre-classification model is trained by: obtaining an original training sample set, which includes original image samples and their category labels, wherein the category labels are configured to indicate whether the image samples contain a target object; performing data augmentation on the original image samples in the original training sample set to obtain an augmented first training sample set; and training the first image pre-classification model using the first training sample set to obtain a trained image pre-classification model.

[0320] In some embodiments, data augmentation includes: performing random data augmentation on image samples based on one or more random data augmentation strategies to obtain augmented image samples; performing data mixing operations on the augmented image samples to obtain mixed image samples; and constructing a first training sample set using the augmented image samples and the mixed image samples.

[0321] In some embodiments, the training process of the image pre-classification model includes: inputting augmented image samples into a first image pre-classification model to obtain an augmented image feature embedding representation output by the feature extraction network of the first image pre-classification model and a classification result output by the classification head; inputting mixed image samples into the first image pre-classification model to obtain a mixed image feature embedding representation output by the feature extraction network of the first image pre-classification model; calculating a first loss function using the classification result and class label; adding gradient perturbation to the augmented image feature embedding representation based on the first loss function to obtain a perturbed image feature embedding representation; calculating a second loss function based on the perturbed image feature embedding representation, the augmented image feature embedding representation, and the mixed image feature embedding representation; and optimizing the weight parameters of the image pre-classification model based on the second loss function to obtain a second image pre-classification model.

[0322] In some embodiments, the weight parameters of the feature extraction network of the second image pre-classification model are frozen, and the weight parameters of the classification head of the second image pre-classification model are optimized to obtain a trained image pre-classification model; or, the classification head of the second image pre-classification model is replaced with a new classification head; the weight parameters of the feature extraction network of the second image pre-classification model are frozen, and the weight parameters of the new classification head of the second image pre-classification model are optimized to obtain a trained image pre-classification model.

[0323] In some embodiments, the image enhancement unit 2501 is configured to perform image quality analysis on the first image, determine one or more image enhancement algorithms corresponding to the first image, and perform image enhancement processing on the first image based on one or more image enhancement algorithms to obtain an enhanced second image.

[0324] In some embodiments, one or more image enhancement algorithms include at least one of the following: trapezoidal correction, surface correction, moiré removal, and deblurring.

[0325] In some embodiments, one or more target objects and their object types are determined by a second device;

[0326] The image information extraction device 250 further includes: a communication unit configured to upload a second image to a second device, so that the second device performs image detection on the second image to determine the location information and object type of a target object in the second image; receive the location information and object type of the target object sent by the second device; and determine the target object in the second image based on the location information of the target object.

[0327] In some embodiments, the attribute information includes the element type of the current element and the position information of the first detection box, and the association information includes the position association information of the first detection box and the second detection box of the current element. The first detection box is the bounding rectangle of the current element, and the second detection box is the bounding rectangle of the associated element of the current element.

[0328] In some embodiments, the position information of the first detection frame includes the first center coordinates and size; the position association information includes the first center coordinates of the first detection frame and the second center coordinates of the second detection frame.

[0329] In some embodiments, the target object is a chart, which includes at least one of the following: mind map, flowchart, pie chart, bar chart;

[0330] Mind maps contain at least one of the following element types: root node, intermediate node, leaf node, and thick node.

[0331] The element types of a flowchart include at least one of the following: process, decision, subprocess, start / end, document, data, condition, flow direction;

[0332] The element types of a pie chart include at least one of the following: icon title, legend, center point, sector point, and text area;

[0333] The element types of a bar chart include at least one of the following: chart title, bar, x-axis name, x-axis value, y-axis name, y-axis value, and text area.

[0334] In some embodiments, attribute information includes at least element text information;

[0335] The extraction model is configured to perform the following steps: detect multiple elements of the target object and obtain the first coordinates and size of the detection boxes of the multiple elements as attribute information of the multiple elements; determine the number of detection boxes of the target object; if the number of detection boxes is greater than or equal to a first threshold, extract text information based on the first coordinates and size of the third detection box to obtain the text information of the third detection box, wherein the third detection box is part or all of the detection boxes in the target object; if the number of detection boxes is less than the first threshold, extract text information from the target object to obtain the text information of the target object; and determine the text information of the fourth detection box based on the text information of the target object and the first coordinates and size of the fourth detection box.

[0336] In some embodiments, the attribute information includes at least one of the following: element identifier, element type, content, style, padding, and border.

[0337] In some embodiments, the extraction model is trained by the following method: obtaining a second training sample set, the second training sample set including a second image sample and its label information, the second image sample containing a target object, the label information including the position information of the first detection box of each element of the target object, the element type, the element identifier, and the associated element identifier; constructing an initial extraction model of the target object; and training the initial extraction model of the target object using the second training sample set to obtain the trained extraction model of the target object.

[0338] In some embodiments, when the target object is a mind map, the mind map extraction model includes: a backbone network, a feature pyramid network, and a detection head; the mind map is input into the backbone network to output a first feature map, the first feature map is input into the feature pyramid network for multi-scale feature extraction to output a second feature map, and the first and second feature maps are input into the detection head to output the element type of each element of the mind map, the position information of the first detection box, and the position information of the second detection box.

[0339] In some embodiments, when the target object is a flowchart, the flowchart extraction model includes an encoder and a decoder;

[0340] The encoder includes a backbone network and a self-attention module. The flowchart is input into the backbone network for multi-scale feature extraction and outputs multi-scale feature maps. The first-scale feature map is input into the self-attention module and outputs a global feature map. The global feature map is fused with the feature maps of other scales to obtain multiple first feature vectors output by the encoder. The decoder outputs the type information of each element of the flowchart, the position information of the first detection box, and the position information of the second detection box from the multiple first feature vectors.

[0341] In practical applications, the aforementioned device can be a first device or a chip applied to a first device. In this application, the device can implement the functions of multiple units through software, hardware, or a combination of both, enabling the device to execute the image information extraction method provided in any of the above embodiments. Furthermore, the technical effects of each technical solution of the device can be referenced to the technical effects of the corresponding technical solutions in the image information extraction method, and this application will not elaborate further on these.

[0342] The first device may include devices such as mobile phones, tablets, laptops, handheld computers, personal digital assistants (PDAs), portable media players (PMPs), navigation devices, wearable devices, smart bracelets, cameras, etc.

[0343] Based on the hardware implementation of each unit in the above-mentioned image information extraction device, this application embodiment also provides a first device, as shown in FIG25. The first device 250 includes: a processor 2501 and a memory 2502 configured to store a computer program capable of running on the processor.

[0344] When the processor 2501 is configured to run a computer program, it executes the method steps described in the foregoing embodiments.

[0345] Of course, in practical applications, as shown in Figure 25, the various components in the first device 250 are coupled together via a bus system 2503. It can be understood that the bus system 2503 is used to achieve communication between these components. In addition to the data bus, the bus system 2503 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 2503 in the figure.

[0346] In practical applications, the aforementioned processor can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field-Programmable Gate Array (FPGA), controller, microcontroller, and microprocessor. It is understood that, for different devices, the electronic device used to implement the above processor functions can also be other types, and the embodiments of this application do not specifically limit this.

[0347] The aforementioned memory can be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, and provides instructions and data to the processor.

[0348] This application also provides a chip, which includes a processor that can call and run computer programs from memory to implement the methods in this application.

[0349] Optionally, the chip may also include a memory. The processor can retrieve and run computer programs from the memory to implement the methods described in the embodiments of this application.

[0350] The memory can be a separate device independent of the processor, or it can be integrated into the processor.

[0351] Optionally, the chip may also include an input interface. The processor can control this input interface to communicate with other devices or chips; specifically, it can acquire information or data sent by other devices or chips.

[0352] Optionally, the chip may also include an output interface. The processor can control this output interface to communicate with other devices or chips; specifically, it can output information or data to other devices or chips.

[0353] Optionally, the chip can be applied to the first device in the embodiments of this application, and the chip can implement the corresponding processes implemented by the first device in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0354] In an exemplary embodiment, this application also provides a computer-readable storage medium, such as a memory including a computer program, which can be executed by a processor of a first device to perform the steps of the aforementioned method.

[0355] This application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any of the methods in this application.

[0356] Optionally, the computer program product can be applied to the first device in the embodiments of this application, and the computer program instructions cause the computer to execute the corresponding processes implemented by the first device in the various methods of the embodiments of this application. For the sake of brevity, they will not be described in detail here.

[0357] This application also provides a computer program.

[0358] Optionally, the computer program can be applied to the first device in the embodiments of this application. When the computer program is run on a computer, it causes the computer to execute the corresponding processes implemented by the first device in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0359] It should be understood that in the embodiments of this application, data such as user information are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0360] It should be understood that the terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items. The expressions “having,” “may have,” “comprising,” and “including,” or “may include” and “may contain” used herein may be used to indicate the presence of a corresponding feature (e.g., an element such as a numerical value, function, operation, or component), but do not exclude the presence of additional features.

[0361] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and are not necessarily used to describe a specific order or sequence. For example, without departing from the scope of this invention, first information may also be referred to as second information, and similarly, second information may also be referred to as first information.

[0362] The technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0363] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatus, and devices can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical, or other forms.

[0364] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0365] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0366] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Industrial applicability

[0367] This application provides an image information extraction method, apparatus, device, and storage medium. The method includes: when a first image to be processed includes a target object, performing image enhancement on the first image to obtain an enhanced second image; determining one or more target objects and their object types included in the second image; determining an extraction model for the target object based on the object type; extracting image information from the target object using the extraction model to obtain attribute information of multiple elements of the target object and the association information between the elements; and converting the attribute information of the multiple elements of the target object and the association information between the elements into a target file format to obtain a target file of the target object. This enables mobile devices to quickly and accurately extract attribute information and association information of elements from various objects in an image and convert it into a standardized target file format, improving the readability and versatility of the file. Furthermore, secondary editing of the target object can also be achieved based on the target file.

Claims

1. An image information extraction method applied to a first device, wherein, The method includes: If the first image to be processed contains the target object, the first image is enhanced to obtain the enhanced second image; Determine one or more target objects and their object types included in the second image; Based on the object type of the target object, determine the extraction model of the target object; The target object is extracted using the extraction model of the target object to obtain the attribute information of multiple elements of the target object and the association information between the elements; The attribute information of the multiple elements of the target object and the association information between the elements are converted into a target file format to obtain the target file of the target object.

2. The method of claim 1, wherein, The method further includes: The contents of the first file are displayed in the user interface of the first application; Obtain the first image from the first file.

3. The method of claim 2, wherein, The step of obtaining the first image from the first file includes: In response to the first image selection operation, the first image is retrieved from the currently displayed content of the first file; or, Determine one or more extractable images from the first file; use the one or more extractable images as the first image to be processed; or... One or more extractable images are determined from the first file; in response to a second image selection operation, the first image is obtained from the one or more extractable images.

4. The method of claim 3, wherein, The method further includes: Determine the type of the first object to be processed; Based on the first object type, one or more extractable images are determined from the first file.

5. The method of claim 1, wherein, The method further includes: In response to the third image selection operation, the camera is activated; The first image captured by the camera is obtained.

6. The method of claim 1, wherein, The method further includes: In response to an image information extraction operation, display one or more object types that can be extracted; In response to the selection operation of the first object type, the steps of the image information extraction method are performed, wherein the object type of the target object belongs to the first object type.

7. The method of claim 1, wherein, The method further includes: Based on the target file of the target object, the extraction result of the target object is displayed in the first display area.

8. The method of claim 7, wherein, The method further includes: The target object is displayed in a second display area, where the first display area and the second display area are different display areas of the same user interface.

9. The method of claim 7, wherein, The method further includes: In response to the insertion operation of the extraction result of the target object, the target file of the target object is inserted into the first file to which the target object belongs, resulting in the second file; Based on the second file, the target object and its extraction results are displayed in the user interface of the first application; or, The method further includes: In response to the replacement operation of the extraction result of the target object, the target object in the first file is replaced with the target file of the target object to obtain the third file; Based on the third file, the extraction results of the target object are displayed on the user interface of the first application.

10. The method of claim 9, wherein, When the first image includes multiple target objects, the method further includes: In response to the selection operation of the target object, determine the first selected target object; Insertion or replacement operations to obtain the extraction results of the first target object.

11. The method of claim 7 or 9, wherein, The method further includes: The extraction results of the target object are edited to obtain a new target file.

12. The method of claim 11, wherein, The step of editing the extraction results of the target object to obtain a new target file includes: In response to an editing operation on the target object, a first editing menu is displayed, the first editing menu including one or more editing objects; In response to a selection operation on the first editable object, a second editable menu of the first editable object is displayed, the second editable menu including one or more editable elements of the first editable object; Based on the second editing menu, the extraction results of the target object are edited using the obtained editing information to obtain a new target file.

13. The method of claim 12, wherein, When the target object is a chart, the one or more editing objects include at least one of the following: data source, chart type, chart element; When the first editing object is a data source, the second editing menu includes a data table, an insertion operation for the data table, and a deletion operation; When the first editable object is a chart type, the second edit menu includes the original chart type and one or more replaceable chart types; When the first editable object is a chart element, the second edit menu includes the original style and element settings of the target object.

14. The method according to claim 13, wherein, When the target object is a pie chart or a bar chart, the element settings include at least one of the following: element display control, element position setting, and element content setting. When the target object is a mind map, the element settings include at least one of the following: add branch item, delete branch item, insert branch element item, delete branch element item, element style setting item, and element content setting item; When the target object is a flowchart, the element settings include at least one of the following: element insertion, element deletion, element type conversion, element content settings, and element connection relationship settings.

15. The method of claim 9, wherein, The method further includes: In response to the first format conversion operation, the target file is converted into a first file format to obtain a fourth file; the fourth file is used to display the target object in the user interface of the second application, and the first file format is a file format supported by the second application; In response to the second format conversion operation, the second or third file is converted to the second file format to obtain the fifth file.

16. The method of claim 1, wherein, The method further includes: If the first image does not include the target object, the first image is determined to be an unextractable image, and the steps of the image information extraction method are skipped.

17. The method of claim 16, wherein, The method further includes: If the first image does not include the target object, a first prompt message is output to inform the user that the first image is an unextractable image.

18. The method of claim 1, wherein, The method further includes: A first image is pre-classified using an image pre-classification model to obtain a classification result. The classification result is used to indicate whether the first image includes the target object. The image pre-classification model includes a feature extraction network and a classification head. The feature extraction network is configured to extract features from the first image to obtain a feature embedding representation. The classification head is configured to perform image classification based on the feature embedding representation to determine whether the first image is the target object.

19. The method of claim 18, wherein, The method further includes: Obtain the original training sample set, which includes original image samples and their category labels, wherein the category labels are used to indicate whether the image samples contain target objects; Data augmentation is performed on the original image samples in the original training sample set to obtain the augmented first training sample set; The first image pre-classification model is trained using the first training sample set to obtain the trained image pre-classification model.

20. The method of claim 19, wherein, The data enhancements include: Based on one or more random data augmentation strategies, the image samples are randomly augmented to obtain augmented image samples. The enhanced image samples are subjected to a data mixing operation to obtain mixed image samples; The first training sample set is constructed using the enhanced image samples and the hybrid image samples.

21. The method of claim 20, wherein, The step of training the first image pre-classification model using the first training sample set to obtain the trained image pre-classification model includes: The enhanced image sample is input into the first image pre-classification model to obtain the enhanced image feature embedding representation output by the feature extraction network of the first image pre-classification model and the classification result output by the classification head; The hybrid image samples are input into the first image pre-classification model to obtain the hybrid image feature embedding representation output by the feature extraction network of the first image pre-classification model; The first loss function is calculated using the classification results and category labels; Based on the first loss function, gradient perturbation is added to the enhanced image feature embedding representation to obtain the perturbed image feature embedding representation; Based on the perturbated image feature embedding representation, the enhanced image feature embedding representation and the hybrid image feature embedding representation, a second loss function is calculated; The weight parameters of the image pre-classification model are optimized based on the second loss function to obtain the second image pre-classification model.

22. The method of claim 21, wherein, The method further includes: Freeze the weight parameters of the feature extraction network of the second image pre-classification model, optimize the weight parameters of the classification head of the second image pre-classification model, and obtain the trained image pre-classification model; or, Replace the classification head of the second image pre-classification model with a new classification head; Freeze the weight parameters of the feature extraction network of the second image pre-classification model, optimize the weight parameters of the new classification head of the second image pre-classification model, and obtain the trained image pre-classification model.

23. The method of claim 1, wherein, The step of enhancing the first image to obtain the enhanced second image includes: Perform image quality analysis on the first image to determine one or more image enhancement algorithms corresponding to the first image; Based on one or more of the image enhancement algorithms, the first image is subjected to image enhancement processing to obtain the enhanced second image.

24. The method of claim 23, wherein, The one or more image enhancement algorithms include at least one of the following: trapezoidal correction, surface correction, moiré removal, and deblurring.

25. The method of claim 1, wherein, The one or more target objects and their object types are determined by a second device; The method further includes: uploading the second image to the second device, so that the second device performs image detection on the second image to determine the location information and object type of the one or more target objects in the second image; Receive the location information and object type of the one or more target objects sent by the second device; Based on the location information of the one or more target objects, the one or more target objects in the second image are determined.

26. The method of claim 1, wherein, The attribute information includes the element type of the current element and the position information of the first detection box. The association information includes the position association information of the first detection box and the second detection box of the current element. The first detection box is the outer rectangle of the current element, and the second detection box is the outer rectangle of the associated element of the current element.

27. The method of claim 26, wherein, The position information of the first detection frame includes the first center coordinates and size; the position association information includes the first center coordinates of the first detection frame and the second center coordinates of the second detection frame.

28. The method of claim 26, wherein, The target object is a chart, which includes at least one of the following: mind map, flowchart, pie chart, bar chart; The element types of the mind map include at least one of the following: root node, intermediate node, leaf node, and thick node; The element types of the flowchart include at least one of the following: process, decision, subprocess, start / end, document, data, condition, flow direction; The element types of the pie chart include at least one of the following: icon title, legend, center point, sector point, and text area; The element types of the bar chart include at least one of the following: icon title, bar, x-axis name, x-axis value, y-axis name, y-axis value, and text area.

29. The method of claim 26, wherein, The attribute information also includes text information; The extraction model is configured to perform the following steps: Multiple elements of the target object are detected, and the first coordinates and size of the detection boxes of the multiple elements are obtained as the attribute information of the multiple elements; Determine the number of detection boxes for the target object; When the number of detection boxes is greater than or equal to the first threshold, text information is extracted based on the first coordinates and size of the third detection box to obtain the text information of the third detection box. The third detection box is some or all of the detection boxes in the target object. When the number of detection boxes is less than the first threshold, text information is extracted from the target object to obtain the text information of the target object; Based on the text information of the target object, and the first coordinates and size of the fourth detection box, the text information of the fourth detection box is determined.

30. The method of claim 1, wherein, The method further includes: Obtain a second training sample set, the second training sample set including a second image sample and its label information, the second image sample containing the target object, and the label information including the position information of the first detection box of each element of the target object, element type, element identifier, and associated element identifier; Construct an initial extraction model for the target object; The initial extraction model of the target object is trained using the second training sample set to obtain the trained extraction model of the target object.

31. The method of claim 1 or 30, wherein, When the target object is a mind map, the extraction model of the mind map includes: a backbone network, a feature pyramid network, and a detection head; The mind map is input into the backbone network to output a first feature map. The first feature map is input into the feature pyramid network for multi-scale feature extraction to output a second feature map. The first feature map and the second feature map are input into the detection head to output the element type of each element of the mind map, the position information of the first detection box, and the position information of the second detection box.

32. The method of claim 1 or 30, wherein, When the target object is a flowchart, the flowchart extraction model includes an encoder and a decoder; The encoder includes a backbone network and a self-attention module. The flowchart is input into the backbone network to extract multi-scale features and output multi-scale feature maps. The first-scale feature map is input into the self-attention module to output a global feature map. The global feature map is fused with feature maps of other scales to obtain multiple first feature vectors output by the encoder. The multiple first feature vector decoders output the type information of each element of the flowchart, the position information of the first detection box, and the position information of the second detection box.

33. An image information extracting apparatus, wherein, The device includes: The image enhancement unit is configured to enhance the first image when the first image to be processed includes a target object, thereby obtaining an enhanced second image; The determining unit is configured to determine one or more target objects and their object types included in the second image; and to determine the extraction model of the target objects based on the object types of the target objects. The extraction unit is configured to extract image information of the target object using the extraction model of the target object, and obtain attribute information of multiple elements of the target object and the association information between the elements; The extraction unit is further configured to convert the attribute information of the plurality of elements of the target object and the association information between the elements into a target file format to obtain the target file of the target object.

34. A first device, wherein, The first device includes: a processor and a memory configured to store computer programs capable of running on the processor. Wherein, when the processor is configured to run the computer program, it performs the steps of the method according to any one of claims 1 to 32.

35. A computer readable storage medium having stored thereon a computer program, wherein, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 32.

Images