File processing method, model training method, electronic device and storage medium

By iteratively training the initial structured parsing model using the training dataset, a target structured parsing model is generated, which solves the problems of low accuracy and efficiency in multimedia file parsing, achieving high-precision and efficient structured parsing and generating editable files.

WO2026237485A1PCT designated stage Publication Date: 2026-11-19ALIBABA (CHINA) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2026/081482
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-12
Filing Date
2026-03-04
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

Existing structured parsing techniques for multimedia files lack the ability to deeply analyze element styles, types, and their structural relationships, resulting in low parsing accuracy and efficiency.

Method used

A target structured parsing model is used to perform structured parsing of multimedia files. The initial structured parsing model is trained using a training dataset to generate a structured representation that describes the structured relationships between multiple target elements, and an editable file is generated based on this representation.

Benefits of technology

It significantly improves the structured parsing capabilities and generation quality of multimedia files, enhances the accuracy of element detection, classification, and structured relationship recognition, and improves the convenience and efficiency of editing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2026081482_19112026_PF_FP_ABST
    Figure CN2026081482_19112026_PF_FP_ABST
Patent Text Reader

Abstract

The present disclosure relates to the technical field of large models and data processing, and relates to a file processing method, a model training method, an electronic device and a storage medium. The file processing method comprises: acquiring a target multimedia file; using a target structured parsing model to perform structured parsing on the target multimedia file, so as to generate a structured representation, wherein the target structured parsing model is generated by using a training data set to train an initial structured parsing model, the training data set comprises a multimedia element set and an image set, which are obtained by parsing a multimedia sample file, and the structured representation is used for describing the structured relationships among a plurality of target elements, which are extracted from target images contained in the target multimedia file; and on the basis of the structured representation, generating an editable file. The present disclosure solves the technical problems in the related art of the parsing precision and parsing efficiency being low during structured parsing of a multimedia file.
Need to check novelty before this filing date? Find Prior Art

Description

Document processing methods, model training methods, electronic devices and storage media Technical Field

[0001] This disclosure relates to the fields of large model technology and data processing technology, and more specifically, to a file processing method, a model training method, an electronic device, and a storage medium. Background Technology

[0002] With the rapid development of information technology, multimedia files such as presentations have become indispensable visualization tools in daily office work and teaching, and the demand for efficient creation and automated processing is increasingly prominent. The widespread application of multimodal large models in fields such as natural language processing and image recognition has provided new possibilities for the intelligent parsing and generation of presentations. Structured parsing involves extracting organized and editable information from unstructured data. However, current technologies for structured parsing multimedia files mainly focus on text and basic layout recognition, and have limitations in parsing complex layouts and fine structures between elements in presentations. While related technologies can identify the position of elements in presentations to some extent, they lack in-depth analytical capabilities regarding element styles, types, and their structural relationships. Therefore, they still have significant shortcomings in handling element granularity and structural integrity, resulting in low accuracy and efficiency in the structured parsing of multimedia files.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This disclosure provides a file processing method, a model training method, an electronic device, and a storage medium to at least solve the technical problems of low parsing accuracy and efficiency in the structured parsing of multimedia files in related technologies.

[0005] According to one aspect of the present disclosure, a file processing method is provided, comprising: acquiring a target multimedia file; performing structured parsing on the target multimedia file using a target structured parsing model to generate a structured representation, wherein the target structured parsing model is generated after training an initial structured parsing model using a training dataset, the training dataset including: a multimedia element set and an image set, the multimedia element set and the image set being obtained by parsing multimedia sample files, the structured representation being used to describe the structured relationships between multiple target elements, the multiple target elements being extracted from target images contained in the target multimedia file; and generating an editable file based on the structured representation.

[0006] According to another aspect of the embodiments of this disclosure, a model training method is also provided, comprising: acquiring a training dataset, wherein the training dataset includes: a multimedia element set and an image set, the multimedia element set and the image set being obtained by parsing multimedia sample files; training an initial structured parsing model using the training dataset to generate a target structured parsing model, wherein the target structured parsing model is used to perform structured parsing on a target multimedia file to generate a structured representation, the structured representation being used to describe the structured relationships between multiple target elements, the multiple target elements being extracted from target images contained in the target multimedia file.

[0007] According to another aspect of the embodiments of this disclosure, a file processing method is also provided, comprising: obtaining an uneditable slideshow; performing structured parsing of the uneditable slideshow using a target structured parsing model to generate a structured representation of the slideshow, wherein the target structured parsing model is generated after training an initial structured parsing model using a training dataset, the training dataset including: a multimedia element set and an image set, the multimedia element set and the image set being obtained by parsing multimedia sample files, the structured representation of the slideshow being used to describe the structured relationships between multiple slideshow elements, the multiple slideshow elements being extracted from the uneditable slideshow; and generating an editable slideshow based on the structured representation.

[0008] According to another aspect of the embodiments of this disclosure, a file processing method is also provided, comprising: obtaining a file processing request through a first application programming interface, wherein the request data carried in the file processing request includes: a target multimedia file; and returning a file processing response through a second application programming interface, wherein the response data carried in the file processing response includes: an editable file, the editable file being generated based on a structured representation, the structured representation being generated after performing structured parsing on the target multimedia file using a target structured parsing model, the target structured parsing model being generated after training an initial structured parsing model using a training dataset, the training dataset including: a multimedia element set and an image set, the multimedia element set and the image set being obtained by parsing multimedia sample files, the structured representation being used to describe the structured relationships between multiple target elements, and the multiple target elements being extracted from target images contained in the target multimedia file.

[0009] According to another aspect of the embodiments of this disclosure, a file processing method is also provided, comprising: acquiring a currently input file processing dialogue request, wherein the request data carried in the file processing dialogue request includes: a target multimedia file; responding to the file processing dialogue request, returning a file processing dialogue response, wherein the information carried in the file processing dialogue response includes: an editable file, the editable file being generated based on a structured representation, the structured representation being generated after performing structured parsing on the target multimedia file using a target structured parsing model, the target structured parsing model being generated after training an initial structured parsing model using a training dataset, the training dataset including: a multimedia element set and an image set, the multimedia element set and the image set being obtained by parsing multimedia sample files, the structured representation being used to describe the structured relationships between multiple target elements, the multiple target elements being extracted from target images contained in the target multimedia file; and displaying the editable file within a graphical user interface.

[0010] According to another aspect of the embodiments of this disclosure, a file processing method is also provided, comprising: displaying a target multimedia file on the operation interface in response to an input command applied to the operation interface; and displaying an editable file on the operation interface in response to a processing command applied to the operation interface; wherein the editable file is generated based on a structured representation, the structured representation being generated by performing structured parsing on the target multimedia file using a target structured parsing model, the target structured parsing model being generated by training an initial structured parsing model using a training dataset, the training dataset including: a multimedia element set and an image set, the multimedia element set and the image set being obtained by parsing multimedia sample files, the structured representation being used to describe the structured relationships between multiple target elements, and the multiple target elements being extracted from target images contained in the target multimedia file.

[0011] According to another aspect of the embodiments of this disclosure, a file processing system is also provided, including: a client for sending a target multimedia file; a server connected to the client for performing structured parsing of the target multimedia file using a target structured parsing model to generate a structured representation, and generating an editable file based on the structured representation, wherein the target structured parsing model is generated after training an initial structured parsing model using a training dataset, the training dataset including: a multimedia element set and an image set, the multimedia element set and the image set being obtained by parsing multimedia sample files, the structured representation being used to describe the structured relationships between multiple target elements, the multiple target elements being extracted from target images contained in the target multimedia file; the client is also used to output the editable file.

[0012] According to another aspect of the present disclosure, a computing device is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of the present disclosure when it runs.

[0013] According to another aspect of the present disclosure, an electronic device is also provided, including: a memory storing an executable program; and a processor connected to the memory via a bus for running the program, wherein the program executes the methods in various embodiments of the present disclosure during runtime.

[0014] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is executed, it controls the device where the computer-readable storage medium is located to perform the methods of the various embodiments of the present disclosure.

[0015] According to another aspect of the embodiments of this disclosure, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of this disclosure.

[0016] According to another aspect of the embodiments of this disclosure, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods of various embodiments of this disclosure.

[0017] According to another aspect of the embodiments of this disclosure, a computer program is also provided, which, when executed by a processor, implements the methods of the various embodiments of this disclosure.

[0018] In this embodiment, a target multimedia file is acquired, and then a target structured parsing model is used to perform structured parsing on the target multimedia file to generate a structured representation. The target structured parsing model is generated after training an initial structured parsing model using a training dataset, which includes a multimedia element set and an image set. The multimedia element set and image set are obtained by parsing multimedia sample files. The structured representation is used to describe the structured relationships between multiple target elements. Multiple target elements are extracted from target images contained in the target multimedia file. Finally, an editable file is generated based on the structured representation, thereby significantly improving the ability to parse the structure of the target multimedia file and the quality of the generated file. Iterative training of the initial structured parsing model using a dedicated training dataset can significantly improve the accuracy of the target structured parsing model in detecting, classifying, and recognizing structured relationships of various elements in the target multimedia file. Based on the parsed structured representation, an editable file can be automatically generated. The editable file retains the structure and style of the target multimedia file while allowing users to more easily modify, add, or delete content, greatly enhancing the convenience and efficiency of editing. Therefore, the file processing method in this embodiment significantly improves the parsing accuracy and efficiency when performing structured parsing of multimedia files, providing a more intelligent and efficient means for creating, editing and understanding multimedia files, thereby solving the technical problem of low parsing accuracy and efficiency when performing structured parsing of multimedia files in related technologies.

[0019] It is worth noting that the above general description and the following detailed description are merely for illustrative and explanatory purposes and do not constitute a limitation thereof. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this disclosure, illustrate exemplary embodiments of the present disclosure and are used to explain the disclosure, but do not constitute an undue limitation of the disclosure. In the drawings:

[0021] Figure 1 is a schematic diagram of an application scenario of a file processing method according to an embodiment of the present disclosure;

[0022] Figure 2 is a flowchart of a document processing method according to an embodiment of the present disclosure;

[0023] Figure 3 is a schematic diagram of a document processing method according to an embodiment of the present disclosure;

[0024] Figure 4 is a schematic diagram of a training data acquisition process according to an embodiment of the present disclosure;

[0025] Figure 5 is a schematic diagram of another document processing method according to an embodiment of the present disclosure;

[0026] Figure 6 is a flowchart of a model training method according to an embodiment of the present disclosure;

[0027] Figure 7 is a flowchart of another document processing method according to an embodiment of the present disclosure;

[0028] Figure 8 is a flowchart of another document processing method according to an embodiment of the present disclosure;

[0029] Figure 9 is a flowchart of another document processing method according to an embodiment of the present disclosure;

[0030] Figure 10 is a flowchart of another document processing method according to an embodiment of the present disclosure;

[0031] Figure 11 is a structural block diagram of a document processing apparatus according to an embodiment of the present disclosure;

[0032] Figure 12 is a structural block diagram of a model training device according to an embodiment of the present disclosure;

[0033] Figure 13 is a structural block diagram of another document processing apparatus according to an embodiment of the present disclosure;

[0034] Figure 14 is a structural block diagram of another document processing apparatus according to an embodiment of the present disclosure;

[0035] Figure 15 is a structural block diagram of another document processing apparatus according to an embodiment of the present disclosure;

[0036] Figure 16 is a structural block diagram of another document processing apparatus according to an embodiment of the present disclosure;

[0037] Figure 17 is a structural block diagram of a computing device according to an embodiment of the present disclosure;

[0038] Figure 18 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0039] To enable those skilled in the art to better understand the present disclosure, the technical solutions of the present disclosure will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present disclosure, and not all embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present disclosure.

[0040] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0041] The technical solution disclosed herein is primarily implemented using large-scale model technology. Here, "large-scale model" refers to a deep learning model with a massive number of parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of parameters. Large-scale models, also known as foundation models, are pre-trained using large-scale unlabeled corpora to produce pre-trained models with hundreds of millions of parameters. These models are adaptable to a wide range of downstream tasks and exhibit good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0042] It should be noted that, in practical applications, large models can be fine-tuned using a small number of samples to adapt them to different tasks. For example, large models can be widely applied in Natural Language Processing (NLP), computer vision, and speech processing. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. Therefore, the main application scenarios for large models include, but are not limited to, digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design. In this embodiment, data processing using a target structured parsing model in a file structured parsing scenario is used as an example for explanation.

[0043] According to embodiments of this disclosure, a file processing method is provided. It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0044] Considering the large number of model parameters in large models and the limited computing resources of mobile terminals, the method provided in this disclosure can be applied to the application scenario shown in Figure 1, but is not limited thereto. In the application scenario shown in Figure 1, the large model is deployed on server 10. Server 10 can connect to one or more client devices 20 via a local area network (LAN), wide area network (WAN), Internet, or other types of data networks. These client devices 20 may include, but are not limited to, smartphones, tablets, laptops, PDAs, personal computers, smart home devices, and in-vehicle devices. Client devices 20 can interact with users through a graphical user interface to invoke the large model, thereby implementing the method provided in this disclosure.

[0045] In this embodiment of the disclosure, the system consisting of a client device and a server can perform the following steps: the client device sends a target multimedia file to the server; the server connects to the client; after receiving the target multimedia file, the server uses a target structured parsing model to perform structured parsing on the target multimedia file to generate a structured representation; and generates an editable file based on the structured representation. The target structured parsing model is generated after training an initial structured parsing model using a training dataset. The training dataset includes a multimedia element set and an image set, which are obtained by parsing multimedia sample files. The structured representation is used to describe the structured relationships between multiple target elements. The multiple target elements are extracted from the target images contained in the target multimedia file. Finally, the client outputs an editable file.

[0046] It should be noted that with the rapid development of high-performance computing units, the methods provided in this disclosure can also be applied to integrated model machines in other application scenarios. In one optional embodiment, the integrated model machine has multiple built-in models. Users can select one model to adjust as needed to obtain their own model. The high-performance computing unit built into the integrated model machine can then directly call the adjusted model to execute the methods provided in this disclosure. In another optional embodiment, the large integrated model machine has a pre-trained model built-in. Therefore, the high-performance computing unit built into the integrated model machine can directly call this model to execute the methods provided in this disclosure.

[0047] Furthermore, when users need to train their own models, they can upload their own datasets via the client. These datasets are then sent to the server, allowing the server to adjust the pre-trained model using the dataset to obtain the user's customized model, which can then be deployed to the production environment. To facilitate users' model adjustment needs, the server provides complete adjustment tools, development frameworks, and processes, supporting multiple adjustment strategies. This allows the adjusted model to better adapt to different application domains and achieve a high degree of customization.

[0048] In the above operating environment, this disclosure provides a file processing method as shown in Figure 2. Figure 2 is a flowchart of a file processing method according to an embodiment of this disclosure. As shown in Figure 2, the method may include the following steps:

[0049] Step S21: Obtain the target multimedia file.

[0050] Step S22: The target structured parsing model is used to perform structured parsing on the target multimedia file to generate a structured representation. The target structured parsing model is generated by training the initial structured parsing model using a training dataset. The training dataset includes a multimedia element set and an image set, which are obtained by parsing multimedia sample files. The structured representation is used to describe the structured relationships between multiple target elements, which are extracted from the target images contained in the target multimedia file.

[0051] Step S23: Generate an editable file based on the structured representation.

[0052] The target multimedia files mentioned above are presentations or portable document formats (PDFs) that require structured parsing and conversion. These files contain visual information and underlying structural relationships, such as images, charts, and text layout elements. By performing structured parsing on the target multimedia files, non-editable files are transformed into an editable format.

[0053] Target multimedia files can be obtained in various ways, including but not limited to: user uploads, web scraping, file conversion, and internal enterprise sharing. Specifically, users can upload target multimedia files through software or platform interfaces. In certain application scenarios, such as research or automated processing of large amounts of publicly available presentations, target multimedia files can be scraped from the internet using web scraping technology, but the information acquisition process must comply with relevant rules. Furthermore, files of other formats can be converted to obtain target multimedia files, which can then be used as input for structured parsing. Within an enterprise or organization, employees or systems can share or transmit target multimedia files to structured parsing tools for structured analysis and standardization of internal documents.

[0054] The aforementioned target structured parsing model is a specifically trained data model whose function is to identify and understand multiple target elements and the structured relationships between them within a target multimedia file. Structured parsing involves extracting meaningful information from the target multimedia file and organizing it into a structured form. This structured representation is typically a machine-readable format (such as JSON) that describes not only the information of each target element itself (such as text content and image features) but also details how the target elements are related and laid out, such as hierarchical relationships, adjacency order, or relative positions.

[0055] During the training of the initial structured parsing model, a training dataset containing a multimedia element set and an image set is required to ensure accurate structured parsing. The training dataset is generated by parsing and annotating multimedia sample files. The multimedia element set records various attributes of the layout elements in the multimedia sample files, while the image set contains the corresponding visualizations. The initial structured parsing model serves as a base model. By continuously learning from examples in the training dataset, the initial structured parsing model gradually adjusts and evolves into the target structured parsing model. During training, the initial structured parsing model learns to identify the features of different types of elements, understand the structured relationships between elements, and apply this knowledge to new and unknown target multimedia files.

[0056] The target elements mentioned above can be extracted from target images contained in the target multimedia file. Specifically, they can be objects with independent functions and meanings extracted from the target multimedia file. Taking a presentation or PDF as an example, target elements include, but are not limited to: text blocks, images, tables, links and hyperlinks, special layout elements, animations and transitions, etc. Specifically, text blocks include any text elements such as titles, subtitles, body text, bulleted lists, quotations, footnotes, etc. Images include illustrations, photographs, icons, charts, graphics, etc. Tables contain structured arrangements of data or information. Lists and bullets include ordered or unordered lists of items. Links and hyperlinks include links to other parts of a webpage, file, or document. Special layout elements include headers, footers, watermarks, background designs, etc. Animations and transitions, although not visible in static images, are animation effects used in dynamic presentations or transitions between slides.

[0057] Once the structured parsing is complete, a structured representation will be generated. This representation describes the structured relationships between multiple target elements and comprehensively depicts the structural layout and element attributes of the target multimedia file. It can be used as input for downstream applications to generate editable files, rebuild presentations, or perform further analysis and processing.

[0058] Structured relationships can refer to the logical, visual, and functional connections between multiple target elements. Specifically, structured relationships can include, but are not limited to, the following types: hierarchical relationships, positional relationships, style relationships, logical connection relationships, and temporal relationships. Hierarchical relationships represent the hierarchical structure between target elements, such as the relationship between main headings and subheadings, or between headings and body text; they are key to understanding the logical framework of a presentation. Positional relationships represent the relative positions of target elements on the page, such as whether a text block is above or below an image, whether target elements are aligned, and the spacing between target elements. Style relationships represent the appearance attributes of target elements, such as font, color, size, margins, and background; consistency in styles helps maintain the visual style and professionalism of the presentation. Logical connection relationships represent the content-related connections between target elements, such as the relationship between items in a list and the list title, or the connection between a citation and source information. Temporal relationships, when dealing with dynamic target elements, include the display order of target elements or the timeline of animations; they are crucial for reconstructing animation effects or slide transitions.

[0059] Structured representation encodes structured relationships in a machine-readable format, such as Lightweight Exchange Format (JSON) or Extensible Markup Language (XML). This not only allows for the identification and location of each individual target element but also enables the understanding of how these elements are combined to form the overall layout and logical flow of a presentation. Structured representation is extremely important for automated editing, template reuse, content understanding, and intelligent generation.

[0060] Furthermore, editable files are generated based on the structured representation. This process involves converting the parsed structured data into a file format that users can edit and modify, such as documents, tables, editable presentations, or web pages. The layout and content of the target multimedia file are reconstructed using multiple target elements contained in the structured representation and the relationships between these elements.

[0061] For example, in the process of generating an editable file, the structured representation is read first, and then each target element is mapped to a corresponding object or region in the editable file based on the information in the structured representation. Using the layout and style information in the structured representation, the file's layout is reconstructed, including setting the size, position, margins, and alignment of target elements to match the visual effect of the target multimedia file. For files containing logical connections (such as the association between lists and headings) or interactive elements (such as hyperlinks and forms), restoration is also required based on the structured representation to ensure that the generated editable file not only looks similar to the target multimedia file but also has complete functionality. All of the above information is applied to the template or frame of the editable file to generate an editable file. Users can open the editable file to further edit text, adjust the layout, modify styles, etc.

[0062] Based on steps S21 to S23 above, the target multimedia file is obtained, and then a target structured parsing model is used to perform structured parsing on the target multimedia file, generating a structured representation. The target structured parsing model is generated after training the initial structured parsing model using a training dataset, which includes a multimedia element set and an image set. The multimedia element set and image set are obtained by parsing multimedia sample files. The structured representation is used to describe the structured relationships between multiple target elements. Multiple target elements are extracted from the target images contained in the target multimedia file. Finally, an editable file is generated based on the structured representation, thereby significantly improving the ability to parse the structure of the target multimedia file and the quality of the generated file. Iterative training of the initial structured parsing model using a dedicated training dataset can significantly improve the accuracy of the target structured parsing model in detecting, classifying, and recognizing structured relationships of various elements in the target multimedia file. Based on the parsed structured representation, an editable file can be automatically generated. The editable file retains the structure and style of the target multimedia file, while allowing users to more easily modify, add, or delete content, greatly enhancing the convenience and efficiency of editing. Therefore, the file processing method in this embodiment significantly improves the parsing accuracy and efficiency when performing structured parsing of multimedia files, providing a more intelligent and efficient means for creating, editing and understanding multimedia files, thereby solving the technical problem of low parsing accuracy and efficiency when performing structured parsing of multimedia files in related technologies.

[0063] The document processing method in the embodiments of this disclosure will be further explained below.

[0064] In an optional embodiment, the target structured parsing model includes: a target detection model and a target structured prediction model. In step S22, the target structured parsing model is used to perform structured parsing on the target multimedia file to generate a structured representation, including:

[0065] Step S221: Use an object detection model to detect elements in the target image to obtain element information of multiple target elements.

[0066] Step S222: Use the target structured prediction model to perform structured prediction on the target image and element information to obtain a structured representation.

[0067] The object detection model described above can be used to analyze target images and identify various target elements contained within them, including text boxes, images, tables, lists, and titles. Element information for target elements can include location information, category information, and style information. Location information is typically represented using a bounding box, which indicates the target element's position and size within the target image; this can be coordinates, width, and height. Category information determines the type of the target element, such as "title," "paragraph," or "image." Style information may include, but is not limited to, visual attributes such as font, color, size, margins, and padding, depending on the design of the object detection model and the training data.

[0068] After element detection, a target structured prediction model is used to further analyze the relationships between target elements and construct a higher-level structured representation. The target structured prediction model can predict hierarchical relationships (e.g., which elements are children of headings), positional relationships (e.g., element alignment), and logical relationships (e.g., the association between lists and headings). Based on the positional and style information of target elements, the target structured prediction model can infer the overall layout and page structure. All element information and predicted structured relationships are integrated to obtain a structured representation. This structured representation can be a tree structure or a graph structure, where each node represents a target element, and connections between nodes represent the structured relationships between multiple target elements.

[0069] The structured representation described above can be encoded in JSON, XML, or other formats to facilitate machine understanding and processing. It not only describes the attributes of individual target elements but also illustrates how these elements are organized together to form the logical and visual framework of a document or presentation.

[0070] Based on the above optional embodiments, by using a target detection model to detect elements in the target image, the element information of multiple target elements is obtained. Then, a target structured prediction model is used to perform structured prediction on the target image and element information to obtain a structured representation. Thus, the element information extracted from the non-editable image can be transformed into an editable, structured form, providing a foundation for subsequent file generation, editing, and automated processing.

[0071] In one optional embodiment, the element information includes: element attributes and element positions of multiple target elements, wherein the element attributes include at least one of the following: element style, element color, element type, and element size.

[0072] The above element styles describe the visual appearance of the target element, such as bold, italic, and underlined text, borders and shadow effects for images, or line styles for tables. The above element colors describe the color information of the target element, such as text color, background color, or icon color. The above element types describe the classification of the target element, such as headings, subheadings, body text, images, charts, tables, lists, shapes, and icons. The above element sizes indicate the size of the target element, usually expressed as width and height, and may include proportional information to ensure adaptability on different devices. The above element positions describe the specific location of the target element on the page, usually expressed as coordinates of a bounding box, such as the coordinates of the top-left and bottom-right corners. This helps to accurately place each target element when rebuilding an editable document, maintaining the original layout.

[0073] Based on the above optional embodiments, by accurately extracting the element attributes and element positions of multiple target elements, the parsing accuracy of structured parsing can be further improved, enabling users to efficiently edit, modify, or redesign documents based on the structured representation obtained from the parsing, thereby effectively improving the content creation effect.

[0074] In an optional embodiment, step S23, generating an editable file based on structured representation includes at least one of the following: generating an editable presentation based on structured representation; generating editable front-end web page code based on structured representation.

[0075] For example, in generating an editable presentation, the structured representation is first read, and then each target element in the structured representation is matched with its corresponding object in the presentation. For instance, a first-level heading in the structured representation will be mapped to a first-level heading style in the presentation; a piece of body text will be placed in a body text box; and an image element will be converted into an image insertion on the slide. Based on the layout and position information of the target elements in the structured representation, the page layout in the presentation is reconstructed, including the reproduction of details such as the alignment, spacing, and margins of the target elements. According to the style information in the structured representation, appropriate fonts, colors, sizes, and other visual styles are applied to each target element. All mapped element, layout, and style information are integrated, and a new presentation is created using the presentation software's application programming interface or scripting language, or an existing template is edited to fit the content and design of the structured representation. The generated presentation is checked to ensure that multiple target elements can be edited individually, such as adjusting text, replacing images, and changing styles, while maintaining the overall layout and visual effect.

[0076] For example, in generating editable front-end webpage code, the structured representation is first read to understand the type, style, position, and hierarchy of each element. The structured relationships between multiple target elements in the representation are then converted into HyperText Markup Language (HTML) tags, and Cascading Style Sheets (CSS) rules are written to define the appearance and layout of the target elements. For instance, a level-one heading can be converted into a heading tag, and an image element can be converted into an image tag, with specific style attributes to control its style. Based on the element hierarchy in the structured representation, a Document Object Model (DOM) tree can also be constructed. The DOM tree is the basic structure of a webpage, determining the nesting relationships and display order between target elements. If the structured representation contains interactive elements such as buttons, links, or forms, corresponding event monitors and application code need to be added to the generated HTML code to enable the expected interactive behavior on the webpage.

[0077] Based on the above optional embodiments, editable documents can be generated efficiently and accurately based on structured representation by generating editable presentations or generating editable front-end web page code based on structured representation.

[0078] In one optional embodiment, the file processing method in this disclosure further includes:

[0079] Based on the preset file engineering parsing method, the multimedia element set and image set are parsed from the multimedia sample file; the initial detection model is trained using the multimedia element set and image set to generate the target detection model.

[0080] The aforementioned preset file engineering parsing method can be a series of processing steps and technical strategies designed for specific types of multimedia sample files. This method systematically parses multimedia sample files and extracts multimedia elements and images. Preset file engineering parsing methods typically combine specific knowledge of file formats with relevant parsing techniques, effectively handling the complex structures of multimedia sample files and extracting useful information. The core of preset file engineering parsing methods lies in understanding the file format and selecting the appropriate parsing tools. For example, for presentation files based on the XML open file format, specific tool libraries can be used to read and parse them.

[0081] For example, in the process of parsing multimedia sample files, the first step is to open the multimedia sample file using an appropriate library or tool, and then traverse each page of the multimedia sample file. For documents such as presentations or PDFs, this requires processing each slide or each page of the document. Various types of multimedia elements, such as text, images, tables, and charts, are extracted from each page to obtain a set of multimedia elements. The extraction process involves parsing the XML or other format data within the multimedia sample file and converting it into easily manipulated objects or data structures. For multimedia elements containing images, the image data needs to be separated and saved as independent image file formats. In addition to the multimedia elements themselves, attributes of each multimedia element, such as position, size, color, font type, and margins, can also be extracted, playing a crucial role in training the initial detection model.

[0082] After parsing and obtaining the multimedia element set and image set, the initial detection model is trained to generate the object detection model. Specifically, the multimedia element set and image set are preprocessed, such as image enhancement, size normalization, and label encoding, to improve the model training effect. Then, model initialization is performed, selecting the basic architecture of the initial detection model and initializing the model parameters. Specifically, initialization can be done using pre-trained model weights to accelerate the training process and improve performance. The initial detection model is trained using data from the multimedia element set and image set. During training, the initial detection model attempts to learn the features of multimedia elements in order to accurately identify and locate multimedia elements from new multimedia files.

[0083] For example, a loss function can be used to measure the difference between the model's predicted results and the actual element positions and types. Then, algorithms such as backpropagation and gradient descent are used to reduce the loss, thereby continuously adjusting the model parameters. After training is complete, the object detection model is saved and deployed to real-world applications, such as integrating it into presentation parsing tools for automated parsing and structured processing.

[0084] Based on the above optional embodiments, by parsing a set of multimedia elements and a set of images from a multimedia sample file according to a preset file engineering parsing method, the initial detection model is then trained using the set of multimedia elements and the set of images to generate a target detection model. The target detection model can accurately detect and locate target elements in the multimedia file, thereby further improving the parsing efficiency when performing structured parsing of the file.

[0085] In one optional embodiment, the file processing method in this disclosure further includes:

[0086] Obtain the pre-annotated structural relationship set corresponding to the multimedia element set; use the multimedia element set, image set, and pre-annotated structural relationship set to train the initial structured prediction model and generate the target structured prediction model.

[0087] Specifically, in the process of obtaining the pre-annotated structural relationship set, a large model can be used to pre-annotate the multimedia element set, or it can be done manually, thereby obtaining the structural relationships between multimedia elements, such as hierarchical relationships, adjacency relationships, and alignment methods. The annotated element structural relationships are saved in a structured format, such as JSON or XML, forming a pre-annotated structural relationship set. Each element structural relationship clearly indicates the relationship type and attributes between multiple multimedia elements. The multimedia element set, image set, and pre-annotated structural relationship set are then integrated. After integration, each data entry contains a sample image, the multimedia elements on the sample image, and their structural relationships.

[0088] Furthermore, the initial structured prediction model is trained using a multimedia element set, an image set, and a pre-annotated structural relationship set. During the training process, a suitable initial structured prediction model can be selected to understand the image content and predict the structural relationships between multimedia elements.

[0089] Based on the above optional embodiments, by obtaining the pre-annotated structural relationship set corresponding to the multimedia element set, and then using the multimedia element set, image set, and pre-annotated structural relationship set to train the initial structured prediction model, a target structured prediction model is generated. This model is then used to automatically analyze and predict the structural relationships of target elements in multimedia files, providing strong support for subsequent tasks such as template generation and content rearrangement.

[0090] In one alternative embodiment, the access permissions for the target image are set to a locked state.

[0091] The operation permissions for the target image are set to a locked state, meaning the target image is uneditable. Typically, when processing target multimedia files, such as presentations or PDFs, they may contain various elements such as images, text, and charts. If the operation permissions for the target image within the target multimedia file are set to a locked state, it means that the user cannot directly edit the attributes of these elements within the file, such as position, size, and color. However, even if the target image is locked, the file processing method in this embodiment can still use structured parsing technology to understand and describe the content of the target image and the relationships between its constituent elements.

[0092] For example, besides target images with locked access permissions, the target structured parsing model can also handle unstructured or semi-structured data sources. Unstructured data typically lacks a predefined format or pattern, such as free text, social media posts, and scanned documents; semi-structured data has some structure but is not strictly in a database format, such as XML and JSON. During structured parsing, unstructured or semi-structured data sources can be text content from presentations, tables or charts in PDFs, paragraphs in documents, etc. The target structured parsing model can receive the above data as input and parse out the implicit structured relationships, thereby generating a structured representation. For example, for a piece of free text, the target structured parsing model can identify headings, subheadings, list items, etc., and construct a hierarchical tree structure; for a complex table, the target structured parsing model can parse out the attributes and relationships of the table's rows, columns, and cells, facilitating subsequent data analysis or visualization. Therefore, the file processing method in this embodiment can broaden the application scope of the target structured parsing model, enabling it not only to process static, locked target images, but also to flexibly parse data from different sources and in different formats, providing a more comprehensive and in-depth information extraction and understanding capability.

[0093] Based on the above optional embodiments, even if the operation permissions of the target image are set to a locked state, restricting direct editing operations, the target structured parsing model still possesses a powerful ability to parse the image content and its structured relationships with other target elements. Furthermore, the target structured parsing model can also process unstructured or semi-structured data, including free text, tables, and charts, transforming them into structured representations, greatly enhancing the flexibility and practicality of its applications in document understanding, content analysis, and other fields.

[0094] Figure 3 is a schematic diagram of a file processing method according to an embodiment of the present disclosure. As shown in Figure 3, a target multimedia file is obtained, and then a target structured parsing model is used to perform structured parsing on the target multimedia file to generate a structured representation. The target structured parsing model is generated after training an initial structured parsing model using a training dataset. The training dataset includes a multimedia element set and an image set, which are obtained by parsing multimedia sample files. The structured representation is used to describe the structured relationship between multiple target elements. Multiple target elements are extracted from target images contained in the target multimedia file. Finally, an editable file is generated based on the structured representation. The editable file has similar visual effects and structural relationships to the target multimedia file, thereby significantly improving the ability to parse the structure of the target multimedia file and the generation quality.

[0095] Figure 4 is a schematic diagram of a training data acquisition process according to an embodiment of the present disclosure. As shown in Figure 4, the target structured parsing model includes a target detection model and a target structured prediction model. Based on a preset file engineering parsing method, a multimedia element set and an image set are parsed from a multimedia sample file. The initial detection model is trained using the multimedia element set and the image set to generate the target detection model. A pre-annotated structural relationship set corresponding to the multimedia element set is obtained, and then the initial structured prediction model is trained using the multimedia element set, the image set, and the pre-annotated structural relationship set to generate the target structured prediction model.

[0096] Figure 5 is a schematic diagram of another file processing method according to an embodiment of the present disclosure. As shown in Figure 5, a target structured parsing model is used to perform structured parsing on a target multimedia file. The target structured parsing model includes a target detection model and a target structured prediction model. The target detection model is used to detect elements in the target image to obtain element information of multiple target elements. Then, the target structured prediction model is used to perform structured prediction on the target image and element information to obtain a structured representation. An editable file is generated based on the structured representation.

[0097] Figure 6 is a flowchart of a model training method according to an embodiment of the present disclosure. As shown in Figure 6, the method may include the following steps:

[0098] Step S61: Obtain the training dataset, which includes a multimedia element set and an image set, obtained by parsing multimedia sample files;

[0099] Step S62: Train the initial structured parsing model using the training dataset to generate the target structured parsing model. The target structured parsing model is used to perform structured parsing on the target multimedia file to generate a structured representation. The structured representation is used to describe the structured relationship between multiple target elements, which are extracted from the target images contained in the target multimedia file.

[0100] Based on steps S61 to S62 above, a training dataset is obtained, including a multimedia element set and an image set, which are obtained by parsing multimedia sample files. The initial structured parsing model is then trained using the training dataset to generate a target structured parsing model. This target structured parsing model is used to perform structured parsing on target multimedia files to generate structured representations. These structured representations describe the structured relationships between multiple target elements, which are extracted from the target images contained in the target multimedia file. This significantly improves the ability to parse and generate structured data from target multimedia files. Iterative training of the initial structured parsing model using a dedicated training dataset significantly improves the accuracy of the target structured parsing model in detecting, classifying, and recognizing structured relationships among elements in target multimedia files. This further significantly improves the parsing accuracy and efficiency when performing structured parsing on multimedia files, providing a more intelligent and efficient means for creating, editing, and understanding multimedia files. This solves the technical problem of low parsing accuracy and efficiency in related technologies when performing structured parsing on multimedia files.

[0101] The model training method in the embodiments of this disclosure will be further described below.

[0102] In an optional embodiment, step S61, obtaining the training dataset includes:

[0103] Based on the preset file engineering parsing method, the multimedia element set and image set are parsed from the multimedia sample file to obtain the training dataset.

[0104] The aforementioned preset file engineering parsing method can be a series of processing steps and technical strategies designed for specific types of multimedia sample files. This method systematically parses multimedia sample files and extracts multimedia elements and images. Preset file engineering parsing methods typically combine specific knowledge of file formats with relevant parsing techniques, effectively handling the complex structures of multimedia sample files and extracting useful information. The core of preset file engineering parsing methods lies in understanding the file format and selecting the appropriate parsing tools. For example, for presentation files based on the XML open file format, specific tool libraries can be used to read and parse them.

[0105] For example, in the process of parsing multimedia sample files, the first step is to open the multimedia sample file using an appropriate library or tool, and then traverse each page of the multimedia sample file. For documents such as presentations or PDFs, this requires processing each slide or each page of the document. Various types of multimedia elements, such as text, images, tables, and charts, are extracted from each page to obtain a set of multimedia elements. The extraction process involves parsing the XML or other format data within the multimedia sample file and converting it into easily manipulated objects or data structures. For multimedia elements containing images, the image data needs to be separated and saved as independent image file formats. In addition to the multimedia elements themselves, attributes of each multimedia element, such as position, size, color, font type, and margins, can also be extracted, playing a crucial role in training the initial detection model.

[0106] Based on the above optional embodiments, by using a preset file engineering parsing method, a set of multimedia elements and a set of images are parsed from a multimedia sample file, which are then used to train the initial structured parsing model, thereby further improving the parsing efficiency when performing structured parsing on the target multimedia file.

[0107] In an optional embodiment, the initial structured parsing model includes: an initial detection model and an initial structured prediction model; the target structured parsing model includes: a target detection model and a target structured prediction model; in step S62, the initial structured parsing model is trained using a training dataset to generate the target structured parsing model, which includes:

[0108] Step S621: Train the initial detection model using a multimedia element set and an image set to generate an object detection model;

[0109] Step S622: The initial structured prediction model is trained using the multimedia element set, the image set, and the pre-annotated structural relationship set corresponding to the image set to generate the target structured prediction model.

[0110] After parsing and obtaining the multimedia element set and image set, the initial detection model is trained to generate the object detection model. Specifically, the multimedia element set and image set are preprocessed, such as image enhancement, size normalization, and label encoding, to improve the model training effect. Then, model initialization is performed, selecting the basic architecture of the initial detection model and initializing the model parameters. Specifically, initialization can be done using pre-trained model weights to accelerate the training process and improve performance. The initial detection model is trained using data from the multimedia element set and image set. During training, the initial detection model attempts to learn the features of multimedia elements in order to accurately identify and locate multimedia elements from new multimedia files.

[0111] For example, a loss function can be used to measure the difference between the model's predicted results and the actual element positions and types. Then, algorithms such as backpropagation and gradient descent are used to reduce the loss, thereby continuously adjusting the model parameters. After training is complete, the object detection model is saved and deployed to real-world applications, such as integrating it into presentation parsing tools for automated parsing and structured processing.

[0112] Specifically, in the process of obtaining the pre-annotated structural relationship set, a large model can be used to pre-annotate the multimedia element set, or it can be done manually, thereby obtaining the structural relationships between multimedia elements, such as hierarchical relationships, adjacency relationships, and alignment methods. The annotated element structural relationships are saved in a structured format, such as JSON or XML, forming a pre-annotated structural relationship set. Each element structural relationship clearly indicates the relationship type and attributes between multiple multimedia elements. The multimedia element set, image set, and pre-annotated structural relationship set are then integrated. After integration, each data entry contains an image, the multimedia elements on the image, and their structural relationships.

[0113] Furthermore, the initial structured prediction model is trained using a multimedia element set, an image set, and a pre-annotated structural relationship set. During the training process, a suitable initial structured prediction model can be selected to understand the image content and predict the structural relationships between multimedia elements.

[0114] Based on the above optional embodiments, the initial detection model is trained using a multimedia element set and an image set to generate a target detection model. Then, the initial structured prediction model is trained using the multimedia element set, the image set, and the pre-annotated structural relationship set corresponding to the image set to generate a target structured prediction model, thereby further improving the parsing efficiency of structured parsing.

[0115] Figure 7 is a flowchart of another document processing method according to an embodiment of the present disclosure. As shown in Figure 7, the method may include the following steps:

[0116] Step S71: Obtain the non-editable slides;

[0117] Step S72: The target structured parsing model is used to perform structured parsing on the uneditable slides to generate a structured representation of the slides. The target structured parsing model is generated by training the initial structured parsing model with a training dataset. The training dataset includes a multimedia element set and an image set, which are obtained by parsing multimedia sample files. The structured representation of the slides is used to describe the structured relationship between multiple slide elements, which are extracted from the uneditable slides.

[0118] Step S73: Generate editable slides based on the structured representation.

[0119] Based on steps S71 to S73 above, by acquiring uneditable slides, a target structured parsing model is used to perform structured parsing on the uneditable slides, generating a structured representation of the slides. The target structured parsing model is generated after training the initial structured parsing model using a training dataset, which includes a multimedia element set and an image set obtained by parsing multimedia sample files. The slide structured representation describes the structured relationships between multiple slide elements, which are extracted from the uneditable slides. Finally, an editable slide is generated based on the structured representation, thus significantly improving the parsing capability and generation quality of the uneditable slides' structure. Iterative training of the initial structured parsing model using a dedicated training dataset significantly improves the accuracy of the target structured parsing model in detecting, classifying, and recognizing structured relationships among elements in the uneditable slides. Based on the parsed structured representation, an editable file can be automatically generated. The editable file retains the structure and style of the uneditable slides while allowing users to more easily modify, add, or delete content, greatly enhancing the convenience and efficiency of editing. Therefore, the file processing method in this embodiment significantly improves the parsing accuracy and efficiency when performing structured parsing of multimedia files, providing a more intelligent and efficient means for creating, editing and understanding multimedia files, thereby solving the technical problem of low parsing accuracy and efficiency when performing structured parsing of multimedia files in related technologies.

[0120] Figure 8 is a flowchart of another document processing method according to an embodiment of the present disclosure. As shown in Figure 8, the method may include the following steps:

[0121] Step S81: Obtain a file processing request through the first application programming interface, wherein the request data carried in the file processing request includes: the target multimedia file;

[0122] Step S82: Return a file processing response through the second application programming interface. The response data carried in the file processing response includes: an editable file, which is generated based on a structured representation. The structured representation is generated by performing structured parsing on the target multimedia file using a target structured parsing model. The target structured parsing model is generated by training an initial structured parsing model using a training dataset. The training dataset includes: a multimedia element set and an image set, which are obtained by parsing multimedia sample files. The structured representation is used to describe the structured relationships between multiple target elements, which are extracted from the target images contained in the target multimedia file.

[0123] Based on steps S81 to S82 above, a file processing request is obtained through a first application programming interface (API), wherein the request data carried in the file processing request includes: a target multimedia file. Then, a file processing response is returned through a second API, wherein the response data carried in the file processing response includes: an editable file, which is generated based on a structured representation. The structured representation is generated after the target multimedia file is structured and parsed using a target structured parsing model. The target structured parsing model is generated after training an initial structured parsing model using a training dataset, which includes: a multimedia element set and an image set, obtained by parsing multimedia sample files. The structured representation is used to describe the structured relationships between multiple target elements, which are extracted from the target images contained in the target multimedia file. This significantly improves the ability to parse the structured data of the target multimedia file and the quality of the generated data. Iterative training of the initial structured parsing model using a dedicated training dataset significantly improves the accuracy of the target structured parsing model in detecting, classifying, and recognizing the structured relationships of various elements in the target multimedia file. Based on the structured representation obtained from the parsing, an editable file can be automatically generated. This editable file retains the structure and style of the target multimedia file, while allowing users to more easily modify, add, or delete content, greatly enhancing the convenience and efficiency of editing. Therefore, the file processing method in this embodiment significantly improves the parsing accuracy and efficiency when performing structured parsing on multimedia files, providing a more intelligent and efficient means for creating, editing, and understanding multimedia files. This solves the technical problem of low parsing accuracy and efficiency in related technologies when performing structured parsing on multimedia files.

[0124] Figure 9 is a flowchart of another document processing method according to an embodiment of the present disclosure. As shown in Figure 9, the method may include the following steps:

[0125] Step S91: Obtain the currently input file processing dialog request, wherein the request data carried in the file processing dialog request includes: the target multimedia file;

[0126] Step S92: In response to the file processing dialog request, a file processing dialog response is returned. The information carried in the file processing dialog response includes: an editable file, which is generated based on a structured representation. The structured representation is generated by performing structured parsing on the target multimedia file using a target structured parsing model. The target structured parsing model is generated by training an initial structured parsing model using a training dataset. The training dataset includes: a multimedia element set and an image set, which are obtained by parsing multimedia sample files. The structured representation is used to describe the structured relationships between multiple target elements, which are extracted from the target images contained in the target multimedia file.

[0127] Step S93: Display the editable file within the graphical user interface.

[0128] Based on steps S91 to S93 above, by acquiring the currently input file processing dialogue request, wherein the request data carried in the file processing dialogue request includes: the target multimedia file, and then responding to the file processing dialogue request, a file processing dialogue reply is returned, wherein the information carried in the file processing dialogue reply includes: an editable file, the editable file is generated based on a structured representation, the structured representation is generated after the target multimedia file is structuredly parsed by the target structured parsing model, the target structured parsing model is generated after training the initial structured parsing model using a training dataset, the training dataset includes: a multimedia element set and an image set, the multimedia element set and the image set are obtained by parsing multimedia sample files, the structured representation is used to describe the structured relationships between multiple target elements, multiple target elements are extracted from the target images contained in the target multimedia file, and finally the editable file is displayed in the graphical user interface, thereby significantly improving the ability to parse the structured data of the target multimedia file and the quality of generation. Iterative training of the initial structured parsing model using a dedicated training dataset can significantly improve the accuracy of the target structured parsing model in detecting, classifying, and recognizing the structured relationships of each element in the target multimedia file. Based on the structured representation obtained from the parsing, an editable file can be automatically generated. This editable file retains the structure and style of the target multimedia file, while allowing users to more easily modify, add, or delete content, greatly enhancing the convenience and efficiency of editing. Therefore, the file processing method in this embodiment significantly improves the parsing accuracy and efficiency when performing structured parsing on multimedia files, providing a more intelligent and efficient means for creating, editing, and understanding multimedia files. This solves the technical problem of low parsing accuracy and efficiency in related technologies when performing structured parsing on multimedia files.

[0129] Figure 10 is a flowchart of another file processing method according to an embodiment of the present disclosure. As shown in Figure 10, the method may include the following steps:

[0130] Step S101: In response to the input command applied to the operation interface, display the target multimedia file on the operation interface;

[0131] Step S102: In response to the processing command applied to the operation interface, an editable file is displayed on the operation interface. The editable file is generated based on a structured representation. This structured representation is generated after the target multimedia file is structured and parsed using a target structured parsing model. The target structured parsing model is generated after training an initial structured parsing model using a training dataset. The training dataset includes a multimedia element set and an image set, which are obtained by parsing multimedia sample files. The structured representation is used to describe the structured relationships between multiple target elements, which are extracted from the target images contained in the target multimedia file. This significantly improves the ability to parse the structure of the target multimedia file and the quality of the generated content. Iterative training of the initial structured parsing model using a dedicated training dataset significantly improves the accuracy of the target structured parsing model in detecting, classifying, and recognizing structured relationships among the elements in the target multimedia file. Based on the parsed structured representation, an editable file is automatically generated. This editable file retains the structure and style of the target multimedia file while allowing users to more easily modify, add, or delete content, greatly enhancing the convenience and efficiency of editing. Therefore, the file processing method in this embodiment significantly improves the parsing accuracy and efficiency when performing structured parsing of multimedia files, providing a more intelligent and efficient means for creating, editing and understanding multimedia files, thereby solving the technical problem of low parsing accuracy and efficiency when performing structured parsing of multimedia files in related technologies.

[0132] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0133] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, because according to this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0134] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. Based on this understanding, the technical solutions of this disclosure, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this disclosure.

[0135] According to embodiments of this disclosure, a file processing apparatus for implementing the above-described file processing method is also provided. FIG11 is a structural block diagram of a file processing apparatus according to an embodiment of this disclosure. As shown in FIG11, the apparatus includes:

[0136] Module 1101 is configured to acquire the target multimedia file;

[0137] The parsing module 1102 is configured to perform structured parsing of the target multimedia file using a target structured parsing model to generate a structured representation. The target structured parsing model is generated by training an initial structured parsing model using a training dataset. The training dataset includes a multimedia element set and an image set, which are obtained by parsing multimedia sample files. The structured representation is used to describe the structured relationships between multiple target elements, which are extracted from the target images contained in the target multimedia file.

[0138] The generation module 1103 is configured to generate editable files based on structured representation.

[0139] Optionally, the parsing module 1102 is further configured to: use an object detection model to perform element detection on the target image to obtain element information of multiple target elements; and use a target structured prediction model to perform structured prediction on the target image and element information to obtain a structured representation.

[0140] Optionally, the element information includes: element attributes and element positions of multiple target elements, wherein the element attributes include at least one of the following: element style, element color, element type, and element size.

[0141] Optionally, the generation module 1103 is also configured to: generate editable presentations based on structured representations; and generate editable front-end web page code based on structured representations.

[0142] Optionally, the parsing module 1102 is further configured to parse the multimedia element set and image set from the multimedia sample file according to a preset file engineering parsing method; the file processing device also includes: a training module 1104, configured to train the initial detection model using the multimedia element set and image set to generate an object detection model.

[0143] Optionally, the acquisition module 1101 is further configured to acquire the pre-annotated structural relationship set corresponding to the multimedia element set; the training module 1105 is further configured to train the initial structured prediction model using the multimedia element set, the image set and the pre-annotated structural relationship set to generate the target structured prediction model.

[0144] Optionally, the access permissions for the target image are set to locked.

[0145] It should be noted that the acquisition module 1101, parsing module 1102, and generation module 1103 correspond to steps S21 to S23 in the above embodiments. The three modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors. The above modules can also be part of a device and run in the server provided in the above embodiments.

[0146] Figure 12 is a structural block diagram of a model training device according to an embodiment of the present disclosure. As shown in Figure 12, the device includes:

[0147] The acquisition module 1201 is configured to acquire a training dataset, which includes a multimedia element set and an image set, obtained by parsing multimedia sample files.

[0148] Training module 1202 is configured to train the initial structured parsing model using the training dataset to generate a target structured parsing model. The target structured parsing model is used to perform structured parsing on the target multimedia file to generate a structured representation. The structured representation is used to describe the structured relationships between multiple target elements, which are extracted from the target images contained in the target multimedia file.

[0149] Optionally, the acquisition module 1201 is also configured to parse the multimedia element set and image set from the multimedia sample file according to a preset file project parsing method to obtain the training dataset.

[0150] Optionally, the training module 1202 is further configured to: train the initial detection model using a multimedia element set and an image set to generate an object detection model; and train the initial structured prediction model using a multimedia element set, an image set, and a pre-labeled structural relationship set corresponding to the image set to generate an object structured prediction model.

[0151] It should be noted that the acquisition module 1201 and training module 1202 correspond to steps S61 to S62 in the above embodiments. The two modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors. The above modules can also be part of a device and run in the server provided in the above embodiments.

[0152] Figure 13 is a structural block diagram of another document processing apparatus according to an embodiment of the present disclosure. As shown in Figure 13, the apparatus includes:

[0153] Module 1301 is configured to retrieve non-editable slides;

[0154] The parsing module 1302 is configured to perform structured parsing of the uneditable slides using a target structured parsing model to generate a structured representation of the slides. The target structured parsing model is generated by training an initial structured parsing model using a training dataset. The training dataset includes a multimedia element set and an image set, which are obtained by parsing multimedia sample files. The structured representation of the slides is used to describe the structured relationships between multiple slide elements, which are extracted from the uneditable slides.

[0155] Module 1303 generates editable slides based on structured representation.

[0156] It should be noted that the acquisition module 1301, parsing module 1302, and generation module 1303 correspond to steps S71 to S73 in the above embodiments. The three modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors. The above modules can also be part of a device and run in the server provided in the above embodiments.

[0157] Figure 14 is a structural block diagram of another document processing apparatus according to an embodiment of the present disclosure. As shown in Figure 14, the apparatus includes:

[0158] The acquisition module 1401 is configured to acquire a file processing request through a first application programming interface, wherein the request data carried in the file processing request includes: a target multimedia file;

[0159] Return module 1402 is configured to return a file processing response via a second application programming interface. The response data carried in the file processing response includes: an editable file, which is generated based on a structured representation. The structured representation is generated by performing structured parsing on the target multimedia file using a target structured parsing model. The target structured parsing model is generated by training an initial structured parsing model using a training dataset. The training dataset includes: a multimedia element set and an image set, which are obtained by parsing multimedia sample files. The structured representation is used to describe the structured relationships between multiple target elements, which are extracted from the target images contained in the target multimedia file.

[0160] It should be noted that the acquisition module 1401 and return module 1402 correspond to steps S811 to S82 in the above embodiments. The two modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors. The above modules can also be part of a device and run in the server provided in the above embodiments.

[0161] Figure 15 is a structural block diagram of another document processing apparatus according to an embodiment of the present disclosure. As shown in Figure 15, the apparatus includes:

[0162] The acquisition module 1501 is configured to acquire the currently input file processing dialog request, wherein the request data carried in the file processing dialog request includes: the target multimedia file;

[0163] Return module 1502 is configured to respond to a file processing dialog request and return a file processing dialog response. The information carried in the file processing dialog response includes: an editable file, which is generated based on a structured representation. The structured representation is generated by performing structured parsing on the target multimedia file using a target structured parsing model. The target structured parsing model is generated by training an initial structured parsing model using a training dataset. The training dataset includes: a multimedia element set and an image set, which are obtained by parsing multimedia sample files. The structured representation is used to describe the structured relationships between multiple target elements, which are extracted from the target images contained in the target multimedia file.

[0164] Display module 1503 is configured to display editable files within a graphical user interface.

[0165] It should be noted that the acquisition module 1501, return module 1502, and display module 1503 correspond to steps S911 to S93 in the above embodiments. The three modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors. The above modules can also be part of a device and run in the server provided in the above embodiments.

[0166] Figure 16 is a structural block diagram of another document processing apparatus according to an embodiment of the present disclosure. As shown in Figure 16, the apparatus includes:

[0167] The first display module 1601 is configured to display the target multimedia file on the operation interface in response to input commands applied to the operation interface.

[0168] The second display module 1602 is configured to display an editable file on the operation interface in response to processing instructions applied to the operation interface.

[0169] The editable file is generated based on a structured representation. The structured representation is generated by parsing the target multimedia file using a target structured parsing model. The target structured parsing model is generated by training an initial structured parsing model using a training dataset. The training dataset includes a multimedia element set and an image set, which are obtained by parsing multimedia sample files. The structured representation is used to describe the structured relationships between multiple target elements, which are extracted from the target images contained in the target multimedia file.

[0170] It should be noted that the first display module 1601 and the second display module 1602 mentioned above correspond to steps S101 to S102 in the above embodiments. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware components or software components stored in memory and processed by one or more processors. The above modules can also be part of a device and run in the server provided in the above embodiments.

[0171] It should be noted that the preferred embodiments involved in the above embodiments of this disclosure are the same as the solutions, application scenarios and implementation processes provided in the above embodiments, but are not limited to the solutions provided in the above embodiments.

[0172] Embodiments of this disclosure provide a file processing system, including: a client for sending a target multimedia file; a server connected to the client for performing structured parsing of the target multimedia file using a target structured parsing model to generate a structured representation, and generating an editable file based on the structured representation. The target structured parsing model is generated after training an initial structured parsing model using a training dataset. The training dataset includes a multimedia element set and an image set, which are obtained by parsing multimedia sample files. The structured representation describes the structured relationships between multiple target elements, which are extracted from target images contained in the target multimedia file. The client is also used to output the editable file.

[0173] Embodiments of this disclosure may provide a computing device. FIG17 is a structural block diagram of a computing device according to an embodiment of the present disclosure. As shown in FIG17, the computing device 170 may include: one or more (only one is shown in the figure) processors 172, memory 174, memory controller, and peripheral interfaces.

[0174] The aforementioned computing device can be understood as an integrated smart terminal, including but not limited to servers, desktop computers, PCs (Personal Computers), all-in-one model machines, etc., and the computing device may have the model described in the above embodiments of this disclosure pre-installed.

[0175] Specifically, this computing device can pre-install various types of models, including but not limited to models in natural language processing, visual processing, speech processing, code processing, and multimodal task processing, thus providing diverse model selection. In different product forms, this computing device can support one or more model usage methods, including but not limited to model training, model invocation, model fine-tuning, model deployment, model inference, and application. In some product forms, this computing device also supports model management, including but not limited to multi-type model management (supporting the management of discriminative, generative, and other model types), model version control (supporting the control of different model versions), and model evaluation (evaluating model performance and effectiveness based on model evaluation tools). In other product forms, this computing device can also create applications based on models, providing API calling capabilities, allowing models to be called into created applications through API interfaces, and providing application management tools for application management and monitoring.

[0176] Furthermore, the computing device may also include data management (supporting the creation and management of model tuning datasets), a training center (providing abundant training resources to help users learn and master AI technology), and basic control capabilities (providing enterprise-level basic control capabilities to ensure the security and efficient operation of the system). Through the above functions, it provides a comprehensive and integrated device for AI development, training, deployment, and application.

[0177] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0178] The processor can invoke an executable program stored in memory via a transmission device to execute the method described in any of the above embodiments.

[0179] Embodiments of this disclosure can provide an electronic device. FIG18 is a structural block diagram of an electronic device according to an embodiment of this disclosure. As shown in FIG18, the electronic device may include: an input / output device 182; a memory 184; and a processor 186, wherein the processor 186 is connected to the input / output device 182 and the memory 184 via a bus 188.

[0180] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0181] The processor can invoke an executable program stored in memory via a transmission device to execute the method described in any of the above embodiments.

[0182] It will be understood by those skilled in the art that the structure shown in the figure is merely illustrative, and the computing device may also be a smartphone, tablet computer, PDA, mobile internet device (MID), PAD, or other terminal device. This figure does not limit the structure of the aforementioned computing device. For example, the computing device may include more or fewer components (such as network interfaces, display devices, etc.) than shown in the figure, or may have a different configuration than that shown in the figure.

[0183] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0184] Embodiments of this disclosure also provide a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store program code executed by the method provided in the above embodiments.

[0185] Optionally, in this embodiment, the storage medium may be located in a computing device.

[0186] Optionally, in this embodiment, the computer-readable storage medium is configured to store an executable program, which, when the executable program is running, controls the device where the computer-readable storage medium is located to execute the method described in any of the above embodiments.

[0187] Embodiments of this disclosure also provide a computer program product. Optionally, in this embodiment, the computer program product may include a computer program that, when executed by a processor, implements the methods provided in the embodiments described above.

[0188] Embodiments of this disclosure also provide a computer program product. Optionally, the computer program product may include a non-volatile computer-readable storage medium, which can be used to store a computer program that, when executed by a processor, implements the methods provided in the embodiments described above.

[0189] Embodiments of this disclosure also provide a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, it implements the method provided in the above embodiments.

[0190] In the above embodiments of this disclosure, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0191] In the several embodiments provided in this disclosure, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0192] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0193] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0194] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0195] The above description is only a preferred embodiment of this disclosure. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles of this disclosure, and these improvements and modifications should also be considered within the scope of protection of this disclosure.

Claims

1. A file processing method, comprising: Obtain the target multimedia file; A target structured parsing model is used to perform structured parsing on target multimedia files to generate a structured representation. The target structured parsing model is generated by training an initial structured parsing model using a training dataset. The training dataset includes a multimedia element set and an image set, which are obtained by parsing multimedia sample files. The structured representation is used to describe the structured relationships between multiple target elements, which are extracted from the target images contained in the target multimedia file. An editable file is generated based on the structured representation.

2. The file processing method according to claim 1, wherein The target structured parsing model includes: a target detection model and a target structured prediction model. The target structured parsing model is used to perform structured parsing on the target multimedia file, generating the structured representation, including: The target detection model is used to perform element detection on the target image to obtain element information of the multiple target elements; The target image and the element information are predicted in a structured manner using the target structured prediction model to obtain the structured representation.

3. The file processing method according to claim 2, wherein, The element information includes: the element attributes and element positions of the plurality of target elements, wherein the element attributes include at least one of the following: element style, element color, element type, and element size.

4. The file processing method according to claim 1, wherein, Generating the editable file based on the structured representation includes at least one of the following: An editable presentation is generated based on the structured representation; Editable front-end web page code is generated based on the structured representation.

5. The file processing method according to claim 2, wherein The file processing method further includes: Based on the preset file project parsing method, the multimedia element set and the image set are parsed from the multimedia sample file; The initial detection model is trained using the multimedia element set and the image set to generate the target detection model.

6. The file processing method according to claim 2 or 5, wherein The file processing method further includes: Obtain the pre-annotated structural relationship set corresponding to the multimedia element set; The initial structured prediction model is trained using the multimedia element set, the image set, and the pre-annotated structural relationship set to generate the target structured prediction model.

7. The file processing method of claim 1, wherein, The access permissions for the target image are set to locked.

8. The file processing method of claim 1, wherein, The target element includes at least one of the following: text block, image, table, link, and hyperlink.

9. The file processing method of claim 1, wherein, The structured relationships between the multiple target elements include at least one of the following: hierarchical relationship, positional relationship, style relationship, logical connection relationship, and time sequence relationship.

10. The file processing method of claim 1, wherein, The structured representation includes a tree structure or a graph structure, where nodes in the tree structure or graph structure represent the target element, and the connection relationships between the nodes represent the structured relationships.

11. A model training method, comprising: Obtain a training dataset, wherein the training dataset includes a multimedia element set and an image set, the multimedia element set and the image set being obtained by parsing multimedia sample files; The initial structured parsing model is trained using the training dataset to generate a target structured parsing model. The target structured parsing model is used to perform structured parsing on the target multimedia file to generate a structured representation. The structured representation is used to describe the structured relationships between multiple target elements, which are extracted from the target images contained in the target multimedia file. 12.The method of Claim 11, wherein, Obtaining the training dataset includes: Based on a preset file engineering parsing method, the multimedia element set and the image set are parsed from the multimedia sample file to obtain the training dataset. 13.The method of Claim 11, wherein, The initial structured parsing model includes an initial detection model and an initial structured prediction model. The target structured parsing model includes an object detection model and a target structured prediction model. The initial structured parsing model is trained using the training dataset to generate the target structured parsing model, which includes: The initial detection model is trained using the multimedia element set and the image set to generate the target detection model; The initial structured prediction model is trained using the multimedia element set, the image set, and the pre-annotated structural relationship set corresponding to the image set to generate the target structured prediction model.

14. A file processing method, comprising: Get non-editable slides; A target structured parsing model is used to perform structured parsing on the uneditable slides to generate a structured representation of the slides. The target structured parsing model is generated by training an initial structured parsing model using a training dataset, which includes a multimedia element set and an image set. The multimedia element set and the image set are obtained by parsing multimedia sample files. The structured representation of the slides is used to describe the structured relationships between multiple slide elements, which are extracted from the uneditable slides. Editable slides are generated based on the structured representation.

15. A document processing method, comprising: A file processing request is obtained through a first application programming interface, wherein the request data carried in the file processing request includes: a target multimedia file; A file processing response is returned via a second application programming interface (API). The response data carried in the file processing response includes: an editable file generated based on a structured representation. This structured representation is generated by performing structured parsing on the target multimedia file using a target structured parsing model. The target structured parsing model is generated by training an initial structured parsing model using a training dataset. The training dataset includes: a multimedia element set and an image set, obtained by parsing multimedia sample files. The structured representation is used to describe the structured relationships between multiple target elements, which are extracted from the target images contained in the target multimedia file.

16. A file processing method, comprising: Obtain the currently input file processing dialog request, wherein the request data carried in the file processing dialog request includes: the target multimedia file; In response to the file processing dialogue request, a file processing dialogue response is returned, wherein the information carried in the file processing dialogue response includes: an editable file, the editable file being generated based on a structured representation, the structured representation being generated after the target multimedia file is structuredly parsed using a target structured parsing model, the target structured parsing model being generated after training an initial structured parsing model using a training dataset, the training dataset including: a multimedia element set and an image set, the multimedia element set and the image set being obtained by parsing multimedia sample files, the structured representation being used to describe the structured relationships between multiple target elements, the multiple target elements being extracted from the target images contained in the target multimedia file; The editable file is displayed within a graphical user interface.

17. A document processing method, comprising: In response to input commands applied to the user interface, the target multimedia file is displayed on the user interface; In response to processing instructions applied to the operation interface, an editable file is displayed on the operation interface; The editable file is generated based on a structured representation, which is generated by parsing the target multimedia file using a target structured parsing model. The target structured parsing model is generated by training an initial structured parsing model using a training dataset. The training dataset includes a multimedia element set and an image set, which are obtained by parsing multimedia sample files. The structured representation is used to describe the structured relationships between multiple target elements, which are extracted from the target images contained in the target multimedia file.

18. A document processing system, comprising: The client is used to send the target multimedia file; The server, connected to the client, is configured to perform structured parsing of a target multimedia file using a target structured parsing model to generate a structured representation, and to generate an editable file based on the structured representation. The target structured parsing model is generated after training an initial structured parsing model using a training dataset. The training dataset includes a multimedia element set and an image set, which are obtained by parsing multimedia sample files. The structured representation describes the structured relationships between multiple target elements, which are extracted from target images contained in the target multimedia file. The client is also used to output the editable file.

19. An electronic device comprising: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the file processing method according to any one of claims 1 to 10, 14 to 17 or the model training method according to any one of claims 11 to 13.

20. A computer readable storage medium comprising a stored executable program, wherein, The computer readable storage medium is controlled to perform the file processing method of any one of claims 1 to 10, 14 to 17 or the model training method of any one of claims 11 to 13 when the executable program is running.