Media information processing method and apparatus, and storage medium

Abstract

Provided are a media information processing method and apparatus, and a storage medium. The method comprises: acquiring a file to be decoded, the file to be decoded comprising a bitstream of media information and presentation control information of the media information, and the presentation control information being used for indicating a presentation mode of the media information; parsing the file to be decoded, and obtaining the bitstream of the media information and the presentation control information of the media information; and, on the basis of the presentation control information, performing decoding processing on the bitstream of the media information, and presenting the media information.

Description

Media information processing methods, devices and storage media

[0001] This disclosure claims priority to Chinese patent application No. 202510068440.9, filed on January 15, 2025, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of audio and video processing technology, and in particular to a media information processing method, apparatus and storage medium. Background Technology

[0003] In traditional media messaging applications, user chats typically use single text, image, or audio / video messages. However, in recently emerging hybrid media messaging applications, users can freely edit and create with text, images, and audio / video, ultimately presenting them together. For example, images and audio can be combined (with audio used as background music). Currently, these new applications often directly combine images and audio into a new video file, which is then transmitted as a media message between different users. This direct conversion and transmission of media content loses the original media information (such as the original images), preventing users from further editing and creating with the original content, resulting in low media presentation flexibility. Summary of the Invention

[0004] On the one hand, a media information processing method is provided, which includes: acquiring a file to be decoded, the file to be decoded including a bitstream of media information and presentation control information of media information, the presentation control information being used to indicate the presentation mode of media information; parsing the file to be decoded to obtain the bitstream of media information and the presentation control information of media information; and decoding the bitstream of media information and presenting media information according to the presentation control information.

[0005] On the other hand, a media information processing method is provided, the method comprising: acquiring media information and presentation control information of the media information, the presentation control information being used to indicate the presentation mode of the media information; encoding the media information to obtain a bitstream of the media information; encapsulating the bitstream of the media information and the presentation control information to obtain a file to be decoded; and sending the file to be decoded.

[0006] On another front, a media information processing apparatus is provided, comprising: an acquisition module for acquiring a file to be decoded, the file including a media information bitstream and media information presentation control information, the presentation control information indicating the presentation mode of the media information; a parsing module for parsing the file to be decoded to obtain the media information bitstream and the media information presentation control information; and a processing module for decoding the media information bitstream and presenting the media information according to the presentation control information.

[0007] On another front, a media information processing apparatus is provided, comprising: an acquisition module for acquiring media information and presentation control information of the media information, the presentation control information indicating the presentation mode of the media information; an encoding module for encoding the media information to obtain a bitstream of the media information; an encapsulation module for encapsulating the bitstream of the media information and the presentation control information to obtain a file to be decoded; and a communication module for sending the file to be decoded.

[0008] In another aspect, a communication device is provided, comprising: a memory and a processor; the memory and the processor are coupled; the memory is used to store computer program instructions executable by the processor; and the processor, when executing the computer program instructions, implements the media information processing method of any of the above embodiments.

[0009] In another aspect, a computer-readable storage medium is provided, on which computer program instructions are stored, which, when executed on a computer (e.g., a communication device or a media information processing device), implement the media information processing method of any of the above embodiments.

[0010] In another aspect, a computer program product is provided, which includes computer program instructions that, when executed, implement the media information processing method of any of the above embodiments. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in this disclosure, the accompanying drawings used in some embodiments of this disclosure will be briefly described below. Obviously, the drawings described below are merely drawings of some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings.

[0012] Figure 1 is a schematic diagram of an optional application scenario of a media information processing method according to some embodiments.

[0013] Figure 2 is a flowchart of a media information processing method according to some embodiments.

[0014] Figure 3 is a flowchart of another media information processing method provided according to some embodiments.

[0015] Figure 4 is a schematic diagram of a media information processing procedure according to some embodiments.

[0016] Figure 5 is a block diagram of a media information processing device according to some embodiments.

[0017] Figure 6 is a block diagram of another media information processing apparatus provided according to some embodiments.

[0018] Figure 7 is a block diagram of a communication device according to some embodiments. Detailed Implementation

[0019] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this disclosure.

[0020] In this disclosure, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. "And / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences.

[0021] It should be noted that in this disclosure, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in this disclosure should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0022] Among the related technologies, those involving joint media presentation mainly include two types: volumetric video playback track groups and preselection group data boxes (denoted as PreselectionGroupBox).

[0023] Here, the presentation orbit group (POTG) is used to indicate the quality level when multiple media components are presented together.

[0024] In media files (or files in the International Organization for Standardization (ISO) Basic Media File Format), when multiple alternative versions of volumetric video components exist, and only certain specific combinations should be played together, volumetric video playback track groups should be used to indicate these combinations for joint playback. It should be understood that the specific implementation of a volumetric video playback track group may vary depending on the application scenario. For example, in 360-degree video, it might be used to indicate how video streams of different resolutions or bitrates are combined to provide a seamless panoramic viewing experience. In virtual reality (VR) applications, it might be used to manage video streams from different perspectives to ensure a smooth visual experience for users moving in different directions.

[0025] For example, the relevant parameters for volumetric video playback track groups are as follows:

[0026] Data box type: 'potg'; a data box used to uniquely identify a volumetric video playback track group within a file.

[0027] Data box container: TrackGroupBox; a trackGroupBox typically contains multiple data boxes associated with a single media track, while the 'potg' data box provides information on how these tracks are presented together as a single experience.

[0028] Mandatory: No. This means that not all ISO basic media file formats are required to include volumetric video playback track groups. It is optional and primarily used for advanced applications that require explicit indication of the quality level of media components during joint presentation.

[0029] Quantity: Zero or more. This means that a media file may have no volumetric video playback track groups, or it may have one or more. Multiple 'potg' data boxes may be used to represent different presentation options or quality levels, allowing the player to dynamically adjust the playback experience based on user selection or system resources.

[0030] The track group type for a playback track is a PlayoutTrackGroupBox (extended from TrackGroupTypeBox in related technologies). A PlayoutTrackGroupBox indicates that the corresponding track belongs to a group of tracks that make up a playback group. For each track in a playback group, its TrackGroupBox must contain a PlayoutTrackGroupBox carrying a unique track group identifier (track_group_id). For all component tracks within a playback track group, their joint quality levels can be selectively defined to indicate different qualities of media content.

[0031] The syntax for volumetric video playback track groups is as follows:

[0032] Semantics:

[0033] A value of 1 for quality_ranking_flag indicates that all component tracks in the volumetric video playback track group have a joint quality level; a value of 0 indicates that none of the component tracks in the volumetric video playback track group have a joint quality level.

[0034] The quality_ranking field indicates the combined quality level of all component tracks within a volumetric video playback track group. The smaller the value of this field, the higher the combined quality level.

[0035] Preselection Group Box corresponds to the concept of preselection in dynamic adaptive streaming over HTTP (DASH) based on the hypertext transfer protocol (HTTP). It is used to indicate how to control the joint presentation of media components when there are multiple different versions of the same content (such as audio tracks of different qualities and video tracks of different qualities). For example, a certain quality of audio and a certain quality of video belong to the same preselection.

[0036] When a track contains a TrackGroupTypeBox whose track group type (denoted as track_group_type) is equal to 'pres', the box is also called a Preselection Group Box, which means that the track contributes to a certain preselection.

[0037] Tracks belonging to all track groups with a track_group_type equal to 'pres' and a specific track group identifier (denoted as track_group_id) are part of the same preselection. This specific track_group_id value is also known as the identifier of the preselection. This means that a preselection is uniquely identified by the track_group_id of its track group.

[0038] When multiple tracks contribute to the same preselection, an optional preselectionprocessing box provides information on how to process the track containing this box within the preselection context and relative to other tracks. Therefore, the content of the preselectionprocessing box may differ for each track within the preselection.

[0039] A preselection consisting of only one track does not require any track-related processing. In this case, the preselection group box typically does not contain a preselectionprocessing box.

[0040] PreselectionProcessingBox, definition:

[0041] Box Type: 'prsp';

[0042] Data container: PreselectionGroupBox;

[0043] Mandatory: No;

[0044] Quantity: zero or one.

[0045] This box contains information on how to process tracks that contribute to the preselection. Boxes for specific media types may be used to describe further processing.

[0046] grammar:

[0047] Semantics:

[0048] track_order: Defines the order of this track relative to other tracks in the pre-selected sequence, as described below.

[0049] sample_merge_flag: When it is equal to 1, it means that this track can be merged with another track, as described below.

[0050] Sample entry-specific specifications may require that pre-selected tracks be provided to the corresponding decoder instance in a specific order. Since other means (such as track identifiers, track_id) are unreliable for this purpose, track_order is used to specify the relative order between pre-selected tracks. A smaller number indicates that, at a given time, the sample on the track containing the sample will be provided to the decoder before the same given sample on other tracks with higher numbers. If the track_order of two pre-selected tracks is set to the same value, or if at least one track is missing a pre-selection processing box, the order of these tracks is not important for pre-selection, and samples can be provided to the decoder in any order.

[0051] A merge group is defined as a set of tracks ordered by track_order, where a track with sample_merge_flag set to 0 is followed by a set of consecutive tracks with sample_merge_flag set to 1. All tracks in a merge group should have the same media type, and all samples should be time-aligned.

[0052] If the sample entry type is associated with a specific codec process used to merge preselected samples, the above process should be used.

[0053] If all tracks in the merge group are of sample entry type "mhm2" (MPEG-H 3D Audio), the merge process is defined in subclause 14.6 of ISO / IEC 23008-3:2019.

[0054] Tracks in a merged group may have different sample entry types.

[0055] If the sample entry type is not associated with a specific codec procedure used to merge preselected samples, the following procedure should be used:

[0056] Merging within a merge group should be performed by forming a tuple of track samples with the same timestamp across contributing tracks. The order of samples within a tuple is determined by `track_order`. These tuples should be formed by concatenating samples at the byte level, resulting in a single sample with the corresponding timestamp. If the goal is to generate a new track, each merge group should generate a single output track conforming to the media type derived from the media type of the merged track.

[0057] For tracks that do not belong to the merge group, this document does not specify a merge process.

[0058] The Preselectiontrackgroupentrybox is defined as follows:

[0059] Box Type: 'prse';

[0060] Data box container: TrackGroupDescriptionBox;

[0061] Mandatory: No;

[0062] Quantity: zero or more.

[0063] Preselected track group entry boxes are used to qualify preselections, for example, by language, genre, or media-specific attributes such as audio rendering instructions, audio interactivity, or channel layout. Attributes indicated in the preselected track group entry box take precedence over those indicated in the contributing track.

[0064] The preselected track group entry box should only describe track groups identified by track_group_type equal to 'prse'.

[0065] All preselected tracks with at least one contributing track and whose track_in_movie flag is set to 1 should be qualified via the preselected track group entry box. Otherwise, the presence of the preselected track group entry box is optional.

[0066] All uniquely qualified pre-selected attributes should exist in the pre-selected pre-selected track group entry box.

[0067] The syntax for the preselected track group entry box is as follows:

[0068] Semantics:

[0069] The information contained in the preselected track group entry box when this preselection is selected is information about the available experience. Suitable boxes for describing a preselection include, but are not limited to, the following list of boxes defined in this document: AudioElementBox, AudioElementSelectionBox, ExtendedLanguageBox, UserDataBox (but here it is meant not to contain any of the other boxes mentioned above), KindBox, LabelBox, AudioRenderingIndicationBox, and ChannelLayout.

[0070] If the UserDataBox is contained within a PreselectionTrackGroupEntryBox, then it should not contain any of the boxes mentioned above.

[0071] The parameter num_tracks specifies the number of non-alternative tracks in the preselected track group group.

[0072] The tracks in the preselected track group are those with a 'pres' track group and whose track_group_id is equal to the identifier (ID) of this preselection.

[0073] The number of non-alternative orbits in the preselected orbit group is the sum of the following:

[0074] The number of orbitals grouped by this preselected orbital group with an alternative_group value of 0, the number of unique non-zero alternative_group values ​​among all orbitals grouped by this preselected orbital group, and the value of the parameter num_tracks should be greater than or equal to the number of non-alternate orbitals grouped by this preselected orbital group in this file. It should be understood that having an alternative_group value of 0 indicates that the number of orbitals grouped by this orbital group is unknown or not important for processing the orbital group.

[0075] In some examples, when preselection is split across multiple files, the value of the parameter num_tracks may be greater than the number of non-alternative tracks in this file that contain PreselectionGroupBoxes with the same track_group_id.

[0076] In some examples, when the number of non-alternative tracks grouped by this preselected track group that the player can access is less than the number indicated by the parameter num_tracks, the player may need to omit tracks grouped by this preselected track group.

[0077] The preselection_tag is a codec-specific value that the playback system can provide to the decoder to uniquely identify one of multiple preselections in the media.

[0078] `selection_priority` is an integer used to declare the priority of a preselection when there are no other distinctions (such as those made through media language). Lower numbers indicate higher priority.

[0079] `segment_order` (if present) specifies the recommended sorting rules for the received preselected segments. `segment_order` can take the values ​​0, 1, 2, or other values. Each value has the following semantics:

[0080] 0: Undefined; that is, the sorting rules for the received preselected segments are not defined.

[0081] 1: Sort by time sequence.

[0082] 2: Complete sorting.

[0083] Other values ​​are retained. If segment_order does not exist, its value should be inferred to be 0.

[0084] In some examples, not all tracks that contribute to the pre-selected playback may be delivered in the same file.

[0085] In some examples, KindBox may utilize a rule (role) scheme defined in the relevant technology, as it provides a commonly used scheme to describe pre-selected features.

[0086] In some examples, the preselected track group entry box carries information about the preselected initial experience in the referenced track. The preselected experience may change during playback of these tracks; for example, the audio language may change during playback. These changes are not affected by the information presented in the preselected track group entry box.

[0087] Further media type-specific boxes can be used to describe pre-selected attributes. The reader can ignore and skip unrecognized boxes.

[0088] In traditional media messaging applications, user chats typically use single text, image, or audio / video messages. However, in recently emerging hybrid media messaging applications, users can freely edit and create with text, images, and audio / video, ultimately presenting them together. For example, images and audio can be combined (with audio used as background music). Currently, these new applications often directly combine images and audio into a new video file, which is then transmitted as a media message between different users. This direct conversion and transmission of media content loses the original media information (such as the original image), lacks scalability, and the operation of combining various media components into a single video file cannot be extended to scenarios such as 3D point cloud media or haptic media. Users cannot perform secondary editing and creation of the original content, resulting in low media presentation flexibility.

[0089] In view of this, the present disclosure provides a media information processing method, which involves obtaining a file to be decoded, the file including a media information bitstream and media information presentation control information, the presentation control information being used to indicate the presentation mode of the media information; parsing the file to be decoded to obtain the media information bitstream and media information presentation control information; and decoding the media information bitstream and presenting the media information according to the presentation control information.

[0090] In this way, the files to be decoded obtained by different nodes include the media information bitstream and the media information presentation control information. This allows different nodes to repeatedly edit and present the media information based on the presentation control information, improving the flexibility of media information presentation. It also avoids the loss of original media information during the media information conversion and transmission process, preventing nodes from being unable to perform secondary editing and creation of the original media information.

[0091] For example, as shown in Figure 1, which is a schematic diagram of an optional application scenario of a media information processing method according to some embodiments, the execution end of the media information processing method includes a data acquisition device 21, a control device 22, and a decoding device 23. The data acquisition device 21 is used to acquire media information and send it to the control device 22. Taking a landscape image as an example, the data acquisition device 21 can be a digital camera, a smartphone, a camera mounted on a drone, etc. These devices typically have high-quality lenses, high-resolution image sensors, and advanced image processing technology, enabling them to capture the delicate textures and rich colors of the landscape. The control device 22 can refer to a computer product with communication and storage functions, such as a smartphone, desktop computer, laptop computer, tablet computer, embedded device, or other intelligent device with communication and storage functions. The decoding device 23 can present the image and perform other corresponding operations. It can be a device with a display screen, including but not limited to devices with liquid crystal displays, organic light-emitting diode displays, e-ink screens (such as the screen used in Kindles), such as smartphones, tablet computers, laptop computers, desktop computers, smart TVs, etc.

[0092] In some embodiments, the acquisition device 21, the control device 22, and the decoding device 23 are physically separate devices. The acquisition device 21 is used to acquire media information and media presentation control information. The presentation control information is used to indicate the presentation mode of the media information. The control device 22 obtains the media information and media presentation control information from the acquisition device 21; encodes the media information to obtain the media information bitstream; and encapsulates the media information bitstream and presentation control information to obtain a file to be decoded. The decoding device 23 obtains the file to be decoded from the control device 22, parses the file to be decoded to obtain the media information bitstream and media presentation control information; and decodes and presents the media information bitstream in the file to be decoded according to the presentation control information.

[0093] It should be noted that in some other embodiments, the acquisition device 21 and the control device 22 can be a first device that is physically integrated, and the decoding device 23 can be another device that is physically separated from the first device; or, the control device 22 and the decoding device 23 can be a second device that is physically integrated, and the acquisition device 21 can be another device that is physically separated from the second device; or, the acquisition device, the control device and the decoding device can be combined to form a third device that is physically integrated.

[0094] It should be noted that Figure 1 is only one possible application scenario diagram of the media information processing method. The number of devices included in Figure 1 and the names of each device are not limited. In addition to the devices shown in Figure 1, the application scenario of the media information processing method can also include other devices.

[0095] The application scenarios of the embodiments disclosed herein are not limited. The system architecture and application scenarios described in the embodiments of this disclosure are for the purpose of more clearly illustrating the technical solutions of the embodiments of this disclosure, and do not constitute a limitation on the technical solutions provided by the embodiments of this disclosure. As those skilled in the art will know, with the evolution of network architecture and the emergence of new service scenarios, the technical solutions provided by the embodiments of this disclosure are also applicable to similar technical problems.

[0096] This disclosure provides a media information processing method. As shown in Figure 2, the method includes the following steps:

[0097] S101. Obtain the file to be decoded.

[0098] Here, the file to be decoded includes the bitstream of media information and the presentation control information of the media information. The presentation control information is used to indicate how the media information is presented. For example, the presentation method includes at least one of the following: presentation order, presentation time, presentation transition effect, presentation dependency, and presentation substitution relationship.

[0099] In some embodiments, the media type of the media information includes at least one of the following: image (picture), audio, video, text, tactile media, and immersive media.

[0100] Here, tactile media refers to immersive media of a tactile type, which are media files that can provide consumers with a sensory experience of touch in the real world. Tactile sensation can include, but is not limited to, one or more of the following: vibratory tactile sensation, kinematic tactile sensation, and electrotactile sensation.

[0101] Immersive media refers to media files that provide immersive content, allowing consumers immersed in that content to experience visual, auditory, and tactile sensory experiences reminiscent of the real world. It encompasses media that offer a three- or six-degree-of-freedom viewing experience, including 360-degree video, point cloud media, multi-view media, and mesh media.

[0102] Media types can also include other types, such as, but not limited to: two-dimensional video media, audio media, volumetric video media, multi-view video media, subtitle media, and volumetric media. Volumetric media refers to three-dimensional content media, such as point cloud media.

[0103] The media information disclosed herein may also be called mixed media information or mixed media message, meaning it can contain multiple media types. Terminals can arbitrarily add media components such as images, audio, video, text, haptic media, and immersive media, and send them as a single mixed media message to other terminals (which can be in any application format such as private messages, chat messages, or postings).

[0104] The presentation control information in this disclosure may also be called presentation control metadata, for example.

[0105] In some embodiments, the presentation of control information includes at least one of the following:

[0106] Presenting related information (presenting related metadata); here, presenting related information is used to indicate the relationship between different media information when they are presented in the media information;

[0107] Presentation order information (presentation order metadata); here, presentation order information is used to indicate the order in which at least one piece of media information must be presented.

[0108] Rendering time information (rendering time metadata); rendering time information is used to indicate when at least one piece of media information needs to be rendered.

[0109] Presentation segment information (presentation segment metadata); presentation segment information is used to indicate the portion of media information that needs to be presented, at least one of the media information.

[0110] Present transition information (present transition metadata); Present transition information is used to indicate the presentation effect at the beginning or end of the media information process;

[0111] Rendering dependency information (rendering dependency metadata); rendering dependency information is used to indicate the second media information that the first media information in the media information needs to depend on when it is rendered.

[0112] Present alternative information (present alternative metadata); Present alternative information is used to indicate that when the first media information cannot be presented in the media information, the first media information can be replaced by the second media information.

[0113] Understandably, the addition and transmission of the aforementioned presentation control information facilitates the terminal's control over the joint presentation of multiple types of media information, and also allows for the flexible addition and independent editing of multiple different types of media information. Here, presentation alternative information helps ensure compatibility with terminals possessing different device capabilities, and provides alternative presentation solutions for devices with limited capabilities.

[0114] The file to be decoded in this disclosure may have other names, such as media file, and this disclosure does not limit it.

[0115] S102. Parse the file to be decoded to obtain the bitstream of media information and the presentation control information of media information.

[0116] In some embodiments, media information may exist in the file to be decoded in the form of tracks (dynamic media data) and data items (static media data).

[0117] Here, a track refers to a collection of media data during the media file encapsulation process. A track consists of multiple time-sequential samples. A media file may contain one or more tracks. For example, a video media file may contain, but is not limited to, video media tracks, audio media tracks, and subtitle media tracks. Specifically, metadata information can also be considered a media type, contained in the media file as metadata tracks. Metadata information is a general term for information related to the presentation of media information. This metadata may include descriptive information about the media content, dependency information on which the media depends, and signaling information related to the presentation of the media content, etc.

[0118] A sample is a unit of encapsulation in the media file encapsulation process. A track consists of many samples. For example, a video media track can consist of many samples. A sample is usually a video frame.

[0119] It is understandable that the rendering control information in the file to be decoded can be stored in the file in various forms.

[0120] In some embodiments, presentation control information is encapsulated into the file to be decoded using ISO base media file format (ISOBMFF) data boxes. Here, ISOBMFF is a media file encapsulation standard, and a typical ISOBMFF file is an MP4 file.

[0121] In some embodiments, presentation control information includes entity groups. Entity groups are used to associate presentation combinations with metadata tracks. A presentation combination includes media information entities of at least one media type to be presented within a preset time period. Media information entities include media tracks or media data items.

[0122] In some embodiments, metadata tracks are used to indicate how the presentation composition is rendered. Here, entity groups include at least one of the following fields:

[0123] The first field used to indicate the identifier of the entity group;

[0124] The second field used to indicate the number of entities within an entity group;

[0125] The third field is used to indicate the identifier of an entity within an entity group;

[0126] The fourth field of the identifier used to indicate the component information of the media information corresponding to the entity within the entity group;

[0127] A fifth field used to indicate whether the fourth field exists.

[0128] In some embodiments, a metadata track includes a sample group, which indicates how media samples or media data items in a media track associated with the metadata track are rendered. Here, the entity group includes at least one of the following fields:

[0129] The first field used to indicate the identifier of an entity within an entity group;

[0130] The second field is used to indicate the starting sample number offset value of the sample that should be presented in the sample group corresponding to the entity within the entity group;

[0131] A third field used to indicate whether the second field exists;

[0132] A fourth field used to determine whether a field exists that indicates the presentation transition effect of a data item or sample in the sample group corresponding to an entity within an entity group when the presentation ends;

[0133] The fifth field is used to indicate the duration of the presentation transition effect of the data item or sample in the sample group corresponding to the entity within the entity group when the presentation ends;

[0134] The sixth field is used to indicate the decay mode of the presentation transition effect of data items or samples in the sample group corresponding to entities within an entity group when the presentation ends.

[0135] The seventh field is used to indicate the dependencies of data items or samples in the sample group corresponding to entities within an entity group when they are presented.

[0136] An eighth field used to indicate whether the seventh field exists;

[0137] The ninth field is used to indicate the identifier of the entity that can replace the data item or sample in the sample group corresponding to the entity within the entity group when it cannot be presented.

[0138] The tenth field is used to indicate whether the ninth field exists.

[0139] In some embodiments, presentation control information includes entity groups; entity groups are used to associate presentation combinations, and presentation combinations include media information entities of at least one media type that need to be presented within a preset time period. The media information entities include media tracks or media data items, and the presentation information of the presentation combination is indicated by the metadata of the presentation combination.

[0140] In some embodiments, presentation control information includes entity groups; entity groups are used to associate presentation combinations, presentation combinations include media information entities of at least one media type to be presented within a preset time period, media information entities include media tracks or media data items, and presentation information of presentation combinations is indicated by information data boxes of presentation combinations, and information data boxes are contained in pre-selected data boxes.

[0141] In some embodiments, an entity group includes at least one of the following fields:

[0142] The first field used to indicate the identifier of the entity group;

[0143] The second field used to indicate the number of entities within an entity group;

[0144] The third field is used to indicate the identifier of an entity within an entity group;

[0145] The fourth field used to indicate the presentation category of entities within an entity group;

[0146] The fifth field is used to indicate the presentation order of different entities within an entity group;

[0147] The sixth field is used to indicate the duration of the delay before an entity in an entity group begins to be rendered;

[0148] The seventh field is used to indicate the starting sample number for the rendering of media tracks within an entity group;

[0149] The eighth field used to indicate the rendering end sample number of the media track within an entity group;

[0150] The ninth field used to indicate the duration of presentation of entities within an entity group;

[0151] The tenth field used to indicate whether the seventh field exists;

[0152] The eleventh field is used to indicate whether the eighth field exists;

[0153] The twelfth field is used to indicate whether the ninth field exists.

[0154] S103. Based on the presentation control information, decode the media information bitstream and present the media information.

[0155] In some embodiments, the media information to be presented in the file to be decoded is decoded according to the presentation control information. The media information to be presented may be a part or all of the content of the media information in the file to be decoded.

[0156] In some embodiments, media information is associated and presented based on presentation association information.

[0157] In some embodiments, the portion of the media information to be presented is presented based on the presentation segment information of the media information.

[0158] In some embodiments, the corresponding media information is presented according to the presentation order information and presentation time information of the media information.

[0159] In some embodiments, the presentation transition effect of media information is controlled based on the presentation transition effect information of the media information.

[0160] In some embodiments, the corresponding media information is presented based on the presentation dependency information between media information.

[0161] In some embodiments, corresponding media information is selectively presented based on presentation substitution information between media information and terminal information.

[0162] It is understandable that when the file to be decoded includes mixed media information of at least one media type, the mixed media that is decoded and presented according to the presentation control information can be a combination of multiple different media types.

[0163] Based on this, the files to be decoded obtained by different nodes include the media information bitstream and the media information presentation control information. This allows different nodes to repeatedly edit and present the media information based on the presentation control information, improving the flexibility of media information presentation. It also prevents nodes from being unable to perform secondary editing and creation of the original media information due to the loss of original media information during the media information conversion and transmission process.

[0164] This disclosure provides another media information processing method. As shown in Figure 3, the method includes the following steps:

[0165] S201. Acquiring media information and media information presentation control information.

[0166] Here, presentation control information is used to indicate how media information is presented. For example, presentation methods include at least one of the following: presentation order, presentation time, presentation transition effects, presentation dependencies, and presentation substitutions.

[0167] In some embodiments, various types of media information are acquired based on user recording and editing operations.

[0168] In some embodiments, the media type of the media information includes at least one of the following: image, audio, video, text, tactile media, and immersive media.

[0169] S202. Encode the media information to obtain the media information bitstream.

[0170] S203. Encapsulate the media information bitstream and presentation control information to obtain the file to be decoded.

[0171] In some embodiments, media information may exist in the file to be decoded in the form of tracks (dynamic media data) and data items (static media data).

[0172] It is understandable that the rendering control information in the file to be decoded can be stored in the file in various forms.

[0173] In some embodiments, presentation control information is encapsulated into the file to be decoded using ISOBMFF data boxes. Here, ISOBMFF is a media file encapsulation standard, and a typical ISOBMFF file is an MP4 file.

[0174] In some embodiments, presentation control information includes entity groups. Entity groups are used to associate presentation combinations with metadata tracks. A presentation combination includes media information entities of at least one media type to be presented within a preset time period. Media information entities include media tracks or media data items.

[0175] In some embodiments, metadata tracks are used to indicate how the presentation composition is rendered. Here, entity groups include at least one of the following fields:

[0176] The first field used to indicate the identifier of the entity group;

[0177] The second field used to indicate the number of entities within an entity group;

[0178] The third field is used to indicate the identifier of an entity within an entity group;

[0179] The fourth field of the identifier used to indicate the component information of the media information corresponding to the entity within the entity group;

[0180] A fifth field used to indicate whether the fourth field exists.

[0181] In some embodiments, a metadata track includes a sample group, which indicates how media samples or media data items in a media track associated with the metadata track are rendered. Here, the entity group includes at least one of the following fields:

[0182] The first field used to indicate the identifier of an entity within an entity group;

[0183] The second field is used to indicate the starting sample number offset value of the sample that should be presented in the sample group corresponding to the entity within the entity group;

[0184] A third field used to indicate whether the second field exists;

[0185] A fourth field used to determine whether a field exists that indicates the presentation transition effect of a data item or sample in the sample group corresponding to an entity within an entity group when the presentation ends;

[0186] The fifth field is used to indicate the duration of the presentation transition effect of the data item or sample in the sample group corresponding to the entity within the entity group when the presentation ends;

[0187] The sixth field is used to indicate the decay mode of the presentation transition effect of data items or samples in the sample group corresponding to entities within an entity group when the presentation ends.

[0188] The seventh field is used to indicate the dependencies of data items or samples in the sample group corresponding to entities within an entity group when they are presented.

[0189] An eighth field used to indicate whether the seventh field exists;

[0190] The ninth field is used to indicate the identifier of the entity that can replace the data item or sample in the sample group corresponding to the entity within the entity group when it cannot be presented.

[0191] The tenth field is used to indicate whether the ninth field exists.

[0192] In some embodiments, presentation control information includes entity groups; entity groups are used to associate presentation combinations, and presentation combinations include media information entities of at least one media type that need to be presented within a preset time period. The media information entities include media tracks or media data items, and the presentation information of the presentation combination is indicated by the metadata of the presentation combination.

[0193] In some embodiments, presentation control information includes entity groups; entity groups are used to associate presentation combinations, presentation combinations include media information entities of at least one media type to be presented within a preset time period, media information entities include media tracks or media data items, and presentation information of presentation combinations is indicated by information data boxes of presentation combinations, and information data boxes are contained in pre-selected data boxes.

[0194] In some embodiments, an entity group includes at least one of the following fields:

[0195] The first field used to indicate the identifier of the entity group;

[0196] The second field used to indicate the number of entities within an entity group;

[0197] The third field is used to indicate the identifier of an entity within an entity group;

[0198] The fourth field used to indicate the presentation category of entities within an entity group;

[0199] The fifth field is used to indicate the presentation order of different entities within an entity group;

[0200] The sixth field is used to indicate the duration of the delay before an entity in an entity group begins to be rendered;

[0201] The seventh field is used to indicate the starting sample number for the rendering of media tracks within an entity group;

[0202] The eighth field used to indicate the rendering end sample number of the media track within an entity group;

[0203] The ninth field used to indicate the duration of presentation of entities within an entity group;

[0204] The tenth field used to indicate whether the seventh field exists;

[0205] The eleventh field is used to indicate whether the eighth field exists;

[0206] The twelfth field is used to indicate whether the ninth field exists.

[0207] S204. Send the file to be decoded.

[0208] In some embodiments, the file to be decoded can be carried on some messages and sent, such as through private messages, chat messages, postings, or any other application format.

[0209] Based on this, the media information bitstream and its presentation control information are encapsulated together into a file to be decoded and transmitted. This allows different nodes to repeatedly edit and present the media information based on the presentation control information, improving the flexibility of media presentation. It avoids the loss of original media information during the conversion and transmission process, preventing nodes from being unable to perform secondary editing and creation of the original media information. Furthermore, it avoids the unnecessary transcoding overhead in the content production stage caused by converting the original media information into a new media format, as is common in related technologies.

[0210] The following describes several implementation methods for encapsulating presentation control information into a file to be decoded using ISOBMFF data boxes.

[0211] In some embodiments, an implementation method based on an ISOBMFF data box is provided to encapsulate presentation control information into a file to be decoded. The syntax and semantic information here can correspond to combinations of various cases described in the above embodiments.

[0212] Here, the reserved field below is a reserved field used for byte alignment of the data box, and its meaning will not be explained again later.

[0213] Presenting a composite entity group (i.e., the entity group mentioned above), defined as follows:

[0214] Data box type: 'prce';

[0215] Contains: GroupsListBox;

[0216] Mandatory: No;

[0217] Quantity: 0, 1 or more.

[0218] Presentation portfolio entity groups are used to associate multiple media tracks or media data items within a presentation portfolio, along with their corresponding metadata tracks. A presentation portfolio is a collection of media data of various media types presented to the user over a period of time. A presentation portfolio contains media information entities of multiple media types, including images, audio, video, text, haptic media, volumetric media, and virtual avatar media.

[0219] A presentation suite may contain multiple media tracks or media data items. The presentation method of the corresponding media tracks or media data items (e.g., presentation order, presentation time, presentation transition effects, presentation dependencies, presentation substitutions) should be indicated by the presentation suite metadata track.

[0220] When a certain type of media content contains multiple media tracks, such as point cloud media containing geometric tracks and attribute tracks, if there is a relationship between different components of the same media content of that media type, then the presentation of the combined entity group only needs to associate the primary component among the multiple components; if there is no relationship between different components of the same media content of that media type, then the presentation of the combined entity group needs to associate all components of that media content.

[0221] grammar:

[0222] Semantics:

[0223] The group_id indicates the identifier of the combined entity group, and the value of this identifier should be different from the identifier value of any other entity group.

[0224] num_entities_in_group indicates the number of identifiers of entities mapped to the presenting combined entity group; in other words, it indicates the number of entities mapped to the presenting combined entity group.

[0225] `entity_id` indicates the identifier of the entity mapped to the presented composite entity group. This identifier can be mapped to a data item, where the identifier value of the data item is equal to the identifier value of the entity; or it can be mapped to a track, where the identifier value of the track is equal to the identifier value of the entity.

[0226] The `content_component_flag` flag is used to determine whether the field indicating the component information of the media information corresponding to the entity within the entity group exists; that is, to determine whether the `component_group_id` field exists. For example, a value of 1 indicates that the field indicating the component information of the media information corresponding to the corresponding entity exists; a value of 0 indicates that the field indicating the component information of the media information corresponding to the corresponding entity does not exist.

[0227] `component_group_id` indicates the identifier of the component information (also known as the media content component identifier) ​​of the corresponding entity's media information. Different components belonging to the same media content should have the same media content component identifier. If different components belonging to the same media content are associated through other defined track groups or entity groups, the value of this field should be equal to the value of the defined track group or entity group.

[0228] Presenting combined metadata tracks (i.e., the metadata tracks mentioned above), defined as:

[0229] Presentation portfolio metadata tracks are used to indicate how media data of a particular media type is presented in a presentation portfolio (e.g., presentation order, presentation time, presentation transition effects, presentation dependencies, presentation substitutions). One or more presentation portfolio metadata tracks can exist in a single file.

[0230] If the media data indicated by the composite metadata track exists in the file as a track, the composite metadata track needs to be associated with one or more media tracks of the media type it indicates through a track reference of type 'cdsc'.

[0231] grammar:

[0232] Class PresentationCompositionSampleEntry()extends MetaDataSampleEntry('prcm'){

[0233] }

[0234] Present a group of combined fragment samples, defined as follows:

[0235] Presentation sample groups are contained within presentation composite metadata tracks and are used to indicate the presentation method (e.g., presentation order, presentation time, presentation transition effects, presentation dependencies, presentation substitutions) of a media sample or a media data item in a media track associated with the presentation composite metadata track. When presenting samples in a media track corresponding to a presentation sample group, the duration of each sample in the sample group should be consistent with the duration of the corresponding sample in the corresponding media track.

[0236] grammar:

[0237] Semantics:

[0238] `entity_id` indicates the identifier of the entity to be presented within the presentation time of the sample corresponding to the sample group. This identifier can be mapped to a data item, in which case the identifier value of the data item is equal to the identifier value of the entity; this identifier can also be mapped to a track, in which case the identifier value of the track is equal to the identifier value of the entity.

[0239] The `sample_index_offset_flag` flag determines whether the field indicating the starting sample index offset of the sample to be presented within the sample group corresponding to the presented entity in the presented entity group exists; that is, it indicates whether the `sample_index_offset` field below exists. For example, a value of 1 indicates that the field indicating the starting sample index offset of the sample to be presented in the presented entity exists; a value of 0 indicates that the field indicating the starting sample index offset of the sample to be presented in the presented entity does not exist. When the presented entity is a data item, or when the starting sample index offset of the sample to be presented in the presented entity track is 0, this field should be set to 0.

[0240] The sample_index_offset indicates the starting sample index offset value of the sample that should be rendered in the rendered entity.

[0241] The `presentation_transition_flag` flag determines whether a field exists that indicates the presentation transition effect of a data item or sample within the sample group corresponding to an entity at the end of the presentation. Specifically, it indicates whether the `transition_duration` and `transition_decline` fields exist. For example, a value of 1 indicates the existence of the field indicating the presentation transition effect of the data item or sample corresponding to the presented entity at the end of the presentation; a value of 0 indicates the absence of the field indicating the presentation transition effect of the data item or sample corresponding to the presented entity at the end of the presentation.

[0242] transition_duration indicates the duration of the transition effect when the data item or sample corresponding to the rendered entity ends, in units of time scale in the file.

[0243] `transition_decline` indicates the decay mode of the rendering transition effect for the data item or sample corresponding to the rendered entity when the rendering ends. A value of 0 indicates linear decay during the rendering transition (e.g., volume gradually decreases); a value of 1 indicates no decay during the rendering transition (e.g., volume remains unchanged).

[0244] The `presentation_dependency_flag` flag determines whether a field exists that indicates the dependency relationship of a data item or sample within a sample group corresponding to an entity in the presentation. Specifically, it indicates whether the `dependent_entity_id` field exists. For example, a value of 1 indicates the existence of the field indicating the dependency relationship of the data item or sample corresponding to the presented entity; a value of 0 indicates the non-existence of the field indicating the dependency relationship of the data item or sample corresponding to the presented entity.

[0245] The dependent_entity_id indicates the identifier of the entity on which the data item or sample corresponding to the entity depends during rendering. When this field exists, the entity corresponding to this field needs to be rendered within the corresponding time period; otherwise, the entity should not be rendered.

[0246] In some examples, the presentation is instance-dependent; the presentation of haptic media may depend on the presentation of the corresponding audio media, such as device vibration corresponding to an explosion sound. In this case, if the audio is not presented, the haptic media should not be presented either.

[0247] The `presentation_accessibility_flag` flag determines whether a field exists that indicates the alternative presentation relationship when a data item or sample in the sample group corresponding to an entity cannot be presented. In other words, it indicates whether the `alt_entity_id` field exists. For example, a value of 1 indicates the existence of the field indicating the alternative presentation relationship when the data item or sample corresponding to the presented entity cannot be presented; a value of 0 indicates the absence of the field indicating the absence of the field.

[0248] alt_entity_id indicates an identifier that can be used as a substitute entity when the data item or sample corresponding to the entity cannot be rendered.

[0249] In some examples, alternative examples are presented, which can be replaced by text, image effects, or audio if the device does not support the presentation of haptic media.

[0250] For example, referring to Figure 4, a schematic diagram of a media information processing process is provided, including the following steps:

[0251] Step 1: Obtain the file to be decoded. The file contains multiple different types of media components (i.e., audio track #1, audio track #3, picture item #5, and picture item #6 in Figure 4) and presentation control information (metadata track #2 and metadata track #4).

[0252] Step 2: Parse the media presentation control information in the media file.

[0253] The presentation of control information includes one or more of the following:

[0254] Information related to the presentation of different types of media; such as entity groups, which define the relationships between different media components and help us understand which audio, images, or videos are presented as a whole.

[0255] Information on the presentation order of a certain media component;

[0256] The presentation time information of a certain media component;

[0257] The presentation of a segment of information for a certain media component.

[0258] This information is distributed across metadata tracks and sample groups, which record in detail when each media component should be presented, for how long, and in what specific segments.

[0259] Step 3: Based on the media presentation control information, decapsulate and decode the corresponding media information, and present the corresponding media information.

[0260] For audio: Based on the order, number and duration of samples in metadata track #2, first present a portion of samples from audio track #1 (i.e., the entity identifier in the sample group is 1), then present a portion of samples from audio track #3 (i.e., the entity identifier in the sample group is 3, and the sample index offset value can be 100).

[0261] For images: Based on the order, number, and duration of the samples in metadata track #4, first present image item #5 (that is, the entity identifier in the sample group is 5), then present image item #6 (that is, the entity identifier in the sample group is 6).

[0262] In some embodiments, an implementation method based on an ISOBMFF data box is provided to encapsulate presentation control information into a file to be decoded. The syntax and semantic information here can correspond to combinations of various cases described in the above embodiments.

[0263] Presenting a composite entity group, defined as follows:

[0264] Data box type: 'prce';

[0265] Contains: GroupsListBox;

[0266] Mandatory: No;

[0267] Quantity: 0, 1 or more.

[0268] Presentation portfolio entity groups are used to associate multiple media tracks or media data items within a presentation portfolio. A presentation portfolio is a collection of media data of various media types presented to the user over a period of time. A presentation portfolio contains media information entities of various media types, including images, audio, video, text, haptic media, volumetric media, and virtual avatar media.

[0269] A presentation composition can contain multiple media tracks or media data items. In this case, the presentation order, presentation time, presentation transition effects, presentation dependencies, and presentation substitution relationships of the corresponding media tracks or media data items should be indicated by the corresponding metadata in the presentation composition entity group.

[0270] When a certain type of media content contains multiple media tracks, such as point cloud media containing geometric tracks and attribute tracks, if there is a relationship between different components of the same media content of that media type, then the presentation of the combined entity group only needs to associate the primary component among the multiple components; if there is no relationship between different components of the same media content of that media type, then the presentation of the combined entity group needs to associate all components of that media content.

[0271] grammar:

[0272] Here, fields (1) and (2) in the syntax correspond to the presentation order in the above embodiment. Different entities are classified according to their presentation categories. Within the same type, the presentation order is established according to the command. When it is the turn of the corresponding order's entity, the corresponding entity begins to be presented after the delay specified by the delay.

[0273] Fields (3), (4), and (5) in the syntax control the presentation time and presentation segment in the above embodiments. These three flags control the fields that indicate which part of an entity's content needs to be presented and for how long.

[0274] Fields (6), (7), and (8) in the syntax control the rendering transitions, rendering dependencies, and rendering substitutions in the corresponding embodiments. These three corresponding fields control some special effects in the rendering process relatively independently.

[0275] Semantics:

[0276] The group_id indicates the identifier of the combined entity group, and the value of this identifier should be different from the identifier value of any other entity group.

[0277] num_entities_in_group indicates the number of identifiers of entities mapped to the presenting combined entity group (that is, the number of entities in the presenting combined entity group).

[0278] `entity_id` indicates the identifier of the entity mapped to the presented composite entity group. This identifier can be mapped to a data item, where the identifier value of the data item is equal to the identifier value of the entity; or it can be mapped to a track, where the identifier value of the track is equal to the identifier value of the entity.

[0279] The `presentation_type` directive indicates the presentation category of the media information entity to be presented. Different media information entities within the same presentation category are presented in a specific order. For a given presentation category, only one media information entity can be presented at a time.

[0280] presentation_order indicates the presentation order of different media information entities under the same presentation_type.

[0281] presentation_start_delay indicates the delay duration before the media information entity in the current presentation order begins to be presented.

[0282] In some examples, two media that are sequentially consecutive may not be sequential in time. For example, after the first audio clip is played, there may be a period of time before the second audio clip is played.

[0283] The `start_sample_index_flag` flag is used to determine whether the field indicating the starting sample number of the media track within an entity group exists; that is, whether the `start_sample_index` field below exists. For example, a value of 1 indicates that the field indicating the starting sample number (`start_sample_index`) of the media track exists; a value of 0 indicates that the field indicating the starting sample number (`start_sample_index`) of the media track does not exist.

[0284] The `end_sample_index_flag` flag is used to determine whether the field indicating the end sample number of the media track within an entity group exists; that is, whether the `end_sample_index` field below exists. For example, a value of 1 indicates that the field indicating the end sample number of the media track exists; a value of 0 indicates that the field indicating the end sample number of the media track does not exist.

[0285] The `presentation_duration_flag` flag is used to determine whether a field indicating the presentation duration of entities within an entity group exists; that is, whether the `presentation_duration` field below exists. For example, a value of 1 indicates that the field indicating the presentation duration of media information entities exists; a value of 0 indicates that the field indicating the presentation duration of media information entities does not exist.

[0286] `start_sample_index` indicates the starting sample index for rendering a media track. When the `entity_id` identifier is mapped to a media track and this field does not exist, the default starting sample is the first sample in the media track.

[0287] `end_sample_index` indicates the end sample index of the media track. When the `entity_id` identifier is mapped to a media track, and neither this field nor `presentation_duration` exists, the default end sample is the last sample in the media track.

[0288] The presentation_duration indicates the duration of presentation of a media information entity, in units of time scale in the file.

[0289] The explanations of content_component_flag, presentation_transition_flag, presentation_dependency_flag, presentation_accessibility_flag, and the corresponding fields controlled by the flag, as well as other fields, can be found in the semantic descriptions of the relevant fields in the above embodiments, and will not be repeated here.

[0290] In some embodiments, an implementation method based on an ISOBMFF data box is provided to encapsulate presentation control information into a file to be decoded. The syntax and semantic information here can correspond to combinations of various cases described in the above embodiments.

[0291] Presenting combined information data boxes, defined as follows:

[0292] Data box type: 'prci';

[0293] Included in: PreselectionProcessingBox or meta;

[0294] Mandatory: No;

[0295] Quantity: 0 or 1.

[0296] The presentation of the combined information data box can be contained within a preselected data box, which explains how to process and present all media information entities in a preselection.

[0297] A presentation suite is a collection of media data of various media types presented to a user over a period of time. A presentation suite includes multiple media types such as images, audio, video, text, haptic media, volumetric media, and virtual avatar media.

[0298] A presentation suite can contain multiple media tracks or media data items. In this case, the presentation order, presentation time, presentation transition effects, and presentation relationships of the corresponding media tracks or media data items should be indicated by the presentation suite's information data box.

[0299] grammar:

[0300] Semantics:

[0301] The num_entities_for_presentation instruction maps to this preselected data box and needs to control the number of identifiers (the number of entities) of the entities to be presented.

[0302] The entity_id indicates the identifier of the entity that is mapped to the pre-selected data box and needs to control the presentation. This identifier can be mapped to a data item, in which case the identifier value of the data item is equal to the identifier value of the entity; the identifier can also be mapped to a track, in which case the identifier value of the track is equal to the identifier value of the entity.

[0303] For the remaining fields, please refer to the semantic descriptions of the relevant fields in the above embodiments or examples; they will not be repeated here.

[0304] In some embodiments, the above scheme can also utilize the sample group tool in the above embodiments to directly add sample groups in the corresponding media tracks, directly indicating the presentation control information of the corresponding samples. That is, some fields in the PresentationCompositionInfoBox data box are moved to the sample group for indication.

[0305] Present a group of combined fragment samples, defined as follows:

[0306] The presentation of combined clip sample groups is contained within a media track and is used to indicate the presentation transition effects and presentation relationships of the corresponding media samples in a media track during presentation.

[0307] grammar:

[0308] The fields presented here in the combined fragment sample group can be referred to the relevant field semantic descriptions in the above embodiments, and will not be repeated here.

[0309] The foregoing primarily describes the solutions of the embodiments of this disclosure from a methodological perspective. The following also illustrates a media information processing apparatus for executing the media information processing methods in any of the above embodiments and their possible implementations. It is understood that, in order to implement the media information processing method, the media information processing apparatus includes hardware structures and / or software modules corresponding to the execution of various functions; those skilled in the art should readily recognize that, in conjunction with the algorithm steps of the examples described in the embodiments of this disclosure, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0310] This disclosure embodiment can divide the media information processing device into functional modules according to the above method embodiment. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one functional module. The integrated module can be implemented in hardware or software. It should be noted that the module division in this disclosure embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods. The following description uses the example of dividing each functional module according to each function.

[0311] Figure 5 is a block diagram of a media information processing apparatus according to some embodiments. The media information processing apparatus 500 includes: an acquisition module 501, a parsing module 502, and a processing module 503.

[0312] The acquisition module 501 is used to acquire the file to be decoded, which includes the bitstream of media information and the presentation control information of media information. The presentation control information is used to indicate the presentation mode of the media information.

[0313] The parsing module 502 is used to parse the file to be decoded to obtain the bitstream of media information and the presentation control information of media information;

[0314] The processing module 503 is used to decode the media information bitstream and present the media information according to the presentation control information.

[0315] In some embodiments, the media type of the media information includes at least one of the following: image, audio, video, text, tactile media, and immersive media.

[0316] In some embodiments, the presentation of control information includes at least one of the following:

[0317] Presenting related information; here, presenting related information is used to indicate the relationship between different media information when they are presented.

[0318] Presentation order information; here, presentation order information is used to indicate the order in which at least one piece of media information should be presented.

[0319] Presentation time information; presentation time information is used to indicate when at least one piece of media information needs to be presented.

[0320] Presenting segment information; Presenting segment information is used to indicate at least one portion of media information that needs to be presented.

[0321] Presenting transition effect information; Presenting transition effect information is used to indicate the presentation effect of the start or end process of media information;

[0322] Presentation dependency information; presentation dependency information is used to indicate the second media information that the first media information in the media information needs to depend on when it is presented.

[0323] Present alternative information; present alternative information is used to indicate that when the first media information cannot be presented in the media information, the first media information can be replaced by the second media information.

[0324] In some embodiments, presentation control information is encapsulated into the file to be decoded using the ISOBMFF data box format of the International Organization for Standardization (ISO) Basic Media File Format.

[0325] In some embodiments, presentation control information includes entity groups; entity groups are used to associate presentation combinations with metadata tracks, and presentation combinations include media information entities of at least one media type that need to be presented within a preset time period, and media information entities include media tracks or media data items.

[0326] In some embodiments, metadata tracks are used to indicate how the presentation composition is rendered; entity groups include at least one of the following fields:

[0327] The first field used to indicate the identifier of the entity group;

[0328] The second field used to indicate the number of entities within an entity group;

[0329] The third field is used to indicate the identifier of an entity within an entity group;

[0330] The fourth field of the identifier used to indicate the component information of the media information corresponding to the entity within the entity group;

[0331] A fifth field used to indicate whether the fourth field exists.

[0332] In some embodiments, the metadata track includes a sample group, which indicates how media samples or media data items in the media track associated with the metadata track are presented; the entity group includes at least one of the following fields:

[0333] The first field used to indicate the identifier of an entity within an entity group;

[0334] The second field is used to indicate the starting sample number offset value of the sample that should be presented in the sample group corresponding to the entity within the entity group;

[0335] A third field used to indicate whether the second field exists;

[0336] A fourth field used to determine whether a field exists that indicates the presentation transition effect of a data item or sample in the sample group corresponding to an entity within an entity group when the presentation ends;

[0337] The fifth field is used to indicate the duration of the presentation transition effect of the data item or sample in the sample group corresponding to the entity within the entity group when the presentation ends;

[0338] The sixth field is used to indicate the decay mode of the presentation transition effect of data items or samples in the sample group corresponding to entities within an entity group when the presentation ends.

[0339] The seventh field is used to indicate the dependencies of data items or samples in the sample group corresponding to entities within an entity group when they are presented.

[0340] An eighth field used to indicate whether the seventh field exists;

[0341] The ninth field is used to indicate the identifier of the entity that can replace the data item or sample in the sample group corresponding to the entity within the entity group when it cannot be presented.

[0342] The tenth field is used to indicate whether the ninth field exists.

[0343] In some embodiments, presentation control information includes entity groups; entity groups are used to associate presentation combinations, and presentation combinations include media information entities of at least one media type that need to be presented within a preset time period. The media information entities include media tracks or media data items, and the presentation information of the presentation combination is indicated by the metadata of the presentation combination.

[0344] In some embodiments, presentation control information includes entity groups; entity groups are used to associate presentation combinations, presentation combinations include media information entities of at least one media type to be presented within a preset time period, media information entities include media tracks or media data items, and presentation information of presentation combinations is indicated by information data boxes of presentation combinations, and information data boxes are contained in pre-selected data boxes.

[0345] In some embodiments, an entity group includes at least one of the following fields:

[0346] The first field used to indicate the identifier of the entity group;

[0347] The second field used to indicate the number of entities within an entity group;

[0348] The third field is used to indicate the identifier of an entity within an entity group;

[0349] The fourth field used to indicate the presentation category of entities within an entity group;

[0350] The fifth field is used to indicate the presentation order of different entities within an entity group;

[0351] The sixth field is used to indicate the duration of the delay before an entity in an entity group begins to be rendered;

[0352] The seventh field is used to indicate the starting sample number for the rendering of media tracks within an entity group;

[0353] The eighth field used to indicate the rendering end sample number of the media track within an entity group;

[0354] The ninth field used to indicate the duration of presentation of entities within an entity group;

[0355] The tenth field used to indicate whether the seventh field exists;

[0356] The eleventh field is used to indicate whether the eighth field exists;

[0357] The twelfth field is used to indicate whether the ninth field exists.

[0358] For a more detailed description of the acquisition module 501, the parsing module 502, and the processing module 503, as well as a more detailed description of each of their technical features and the beneficial effects, please refer to the corresponding method embodiment section above, which will not be repeated here.

[0359] Figure 6 is a block diagram of another media information processing apparatus according to some embodiments. The media information processing apparatus 600 includes: an acquisition module 601, an encoding module 602, an encapsulation module 603, and a communication module 604.

[0360] Here, module 601 is used to acquire media information and media information presentation control information. The presentation control information is used to indicate the presentation method of the media information.

[0361] Encoding module 602 is used to encode media information to obtain the media information bitstream;

[0362] The encapsulation module 603 is used to encapsulate the bitstream of media information and presentation control information to obtain the file to be decoded.

[0363] Communication module 604 is used to send the file to be decoded.

[0364] For a more detailed description of the acquisition module 601, encoding module 602, encapsulation module 603 and communication module 604, as well as a more detailed description of each of their technical features and beneficial effects, please refer to the corresponding method embodiment section above, which will not be repeated here.

[0365] It should be noted that the modules in Figures 5 and 6 can also be called units; for example, a communication module can be called a communication unit. Furthermore, in the embodiments shown in Figures 5 and 6, the names of the modules may not be those shown in the figures. For example, a communication module can also be called a sending module or a receiving module, and a parsing module can also be called a decapsulation module.

[0366] If the units or modules in Figures 5 and 6 are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this disclosure, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this disclosure. Storage media for storing computer software products include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0367] In the case of implementing the functions of the integrated modules described above in hardware, embodiments of this disclosure also provide a possible structure for a communication device used to execute the media information processing method provided in embodiments of this disclosure. As shown in FIG7, the communication device 700 includes: a communication interface 703, a processor 702, and a bus 704. In some embodiments, the communication device may further include a memory 701.

[0368] Processor 702 may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with embodiments of this disclosure. Processor 702 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with embodiments of this disclosure. Processor 702 may also be a combination that implements computing functions, such as including one or more microprocessor combinations, a combination of a digital signal processor (DSP) and a microprocessor, etc.

[0369] The communication interface 703 is used to connect to other devices via a communication network. This communication network can be Ethernet, wireless access network, wireless local area network (WLAN), etc.

[0370] The memory 701 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0371] In some embodiments, the memory 701 may exist independently of the processor 702. The memory 701 may be connected to the processor 702 via a bus 704 and is used to store instructions or program code. When the processor 702 calls and executes the instructions or program code stored in the memory 701, it can implement the media information processing method provided in the embodiments of this disclosure.

[0372] In other embodiments, the memory 701 may also be integrated with the processor 702.

[0373] Bus 704 can be an extended industry standard architecture (EISA) bus, etc. Bus 704 can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in Figure 7, but this does not mean that there is only one bus or one type of bus.

[0374] Some embodiments of this disclosure provide a computer-readable storage medium (e.g., a non-transitory computer-readable storage medium) storing computer program instructions that, when executed on a computer, cause the computer to perform a media information processing method as described in any of the above embodiments.

[0375] In one exemplary embodiment, the computer may be the aforementioned media information processing device, and this disclosure does not limit the specific form of the computer.

[0376] In some examples, the aforementioned computer-readable storage media may include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks, or magnetic tapes), optical discs (e.g., compact disks (CDs), digital versatile disks (DVDs), etc.), smart cards, and flash memory devices (e.g., erasable programmable read-only memory (EPROMs), cards, sticks, or key drives, etc.). The various computer-readable storage media described in this disclosure may represent one or more devices for storing information and / or other machine-readable storage media. The term "machine-readable storage media" may include, but is not limited to, wireless channels and various other media capable of storing, containing, and / or carrying instructions and / or data.

[0377] This disclosure provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the media information processing method described in any of the above embodiments.

[0378] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any changes or substitutions within the technical scope disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A media information processing method, wherein, The method includes: Obtain the file to be decoded, the file to be decoded including the bitstream of media information and the presentation control information of the media information, the presentation control information being used to indicate the presentation mode of the media information; The file to be decoded is parsed to obtain the bitstream of the media information and the presentation control information of the media information; Based on the presentation control information, the bitstream of the media information is decoded and the media information is presented.

2. The method according to claim 1, wherein, The media type of the media information includes at least one of the following: image, audio, video, text, tactile media, and immersive media.

3. The method according to claim 1, wherein, The presentation control information includes at least one of the following: Presenting related information; wherein, the presented related information is used to indicate the relationship between different media information when they are presented; Presentation order information; wherein, the presentation order information is used to indicate the order in which at least one piece of media information needs to be presented; Presentation time information; the presentation time information is used to indicate the time when at least one piece of media information needs to be presented. Presentation segment information; the presentation segment information is used to indicate at least one portion of the media information that needs to be presented. Presenting transition effect information; the presenting transition effect information is used to indicate the presentation effect of the start or end process of the media information; Presentation dependency information; the presentation dependency information is used to indicate the second media information that the first media information needs to depend on when it is presented. Present alternative information; the presentation alternative information is used to indicate that when the first media information in the media information cannot be presented, the first media information can be replaced by the second media information.

4. The method according to claim 1, wherein, The presentation control information is encapsulated into the file to be decoded using the ISOBMFF data box format of the International Organization for Standardization (ISO).

5. The method according to claim 4, wherein, The presentation control information includes entity groups; the entity groups are used to associate presentation combinations and metadata tracks, the presentation combinations include media information entities of at least one media type that need to be presented within a preset time period, and the media information entities include media tracks or media data items.

6. The method according to claim 5, wherein, The metadata track is used to indicate the presentation method of the presentation combination; the entity group includes at least one of the following fields: The first field used to indicate the identifier of the entity group; A second field used to indicate the number of entities within the entity group; The third field used to indicate the identifier of an entity within the entity group; The fourth field is an identifier used to indicate the component information of the media information corresponding to the entity within the entity group; A fifth field used to indicate whether the fourth field exists.

7. The method according to claim 5, wherein, The metadata track includes sample groups, which are used to indicate how media samples or media data items in the media track associated with the metadata track are presented; The entity group includes at least one of the following fields: The first field is used to indicate the identifier of an entity within the entity group; The second field is used to indicate the starting sample number offset value of the sample that should be presented in the sample group corresponding to the entity in the entity group; A third field used to indicate whether the second field exists; A fourth field used to determine whether a field exists that indicates the presentation transition effect of a data item or sample in the sample group corresponding to an entity within the entity group when the presentation ends; The fifth field is used to indicate the duration of the presentation transition effect of the data item or sample in the sample group corresponding to the entity within the entity group when the presentation ends; The sixth field is used to indicate the decay mode of the presentation transition effect of data items or samples in the sample group corresponding to the entity within the entity group when the presentation ends. The seventh field is used to indicate the dependencies of data items or samples in the sample group corresponding to the entity within the entity group when they are presented. An eighth field used to indicate whether the seventh field exists; The ninth field is used to indicate the identifier of the entity that can replace the data item or sample in the sample group corresponding to the entity within the entity group when it cannot be presented. A tenth field used to indicate whether the ninth field exists.

8. The method according to claim 4, wherein, The presentation control information includes entity groups; the entity groups are used to associate presentation combinations, the presentation combinations include media information entities of at least one media type that need to be presented within a preset time period, the media information entities include media tracks or media data items, and the presentation information of the presentation combination is indicated by the metadata of the presentation combination.

9. The method according to claim 4, wherein, The presentation control information includes entity groups; the entity groups are used to associate presentation combinations, the presentation combinations include media information entities of at least one media type that need to be presented within a preset time period, the media information entities include media tracks or media data items, the presentation information of the presentation combination is indicated by the information data box of the presentation combination, and the information data box is contained in a pre-selected data box.

10. The method according to claim 8 or 9, wherein, The entity group includes at least one of the following fields: The first field used to indicate the identifier of the entity group; A second field used to indicate the number of entities within the entity group; The third field used to indicate the identifier of an entity within the entity group; The fourth field used to indicate the presentation category of entities within the entity group; The fifth field is used to indicate the presentation order of different entities within the entity group; The sixth field is used to indicate the duration of the delay before an entity in the entity group begins to be rendered; The seventh field used to indicate the presentation start sample number of the media track within the entity group; The eighth field used to indicate the presentation end sample number of the media track within the entity group; A ninth field used to indicate the presentation duration of entities within the entity group; The tenth field used to indicate whether the seventh field exists; The eleventh field used to indicate whether the eighth field exists; The twelfth field is used to indicate whether the ninth field exists.

11. A media information processing method, wherein, The method includes: Acquire media information and presentation control information of the media information, wherein the presentation control information is used to indicate the presentation mode of the media information; The media information is encoded to obtain the bitstream of the media information; The media information bitstream and the presentation control information are encapsulated to obtain the file to be decoded; Send the file to be decoded.

12. The method according to claim 11, wherein, The media type of the media information includes at least one of the following: image, audio, video, text, tactile media, and immersive media.

13. The method according to claim 11, wherein, The presentation control information includes at least one of the following: Presenting related information; the presented related information is used to indicate the relationship between different media information when they are presented in the media information; Presenting sequence information; The presentation order information is used to indicate the order in which at least one piece of media information needs to be presented; Present time information; The presentation time information is used to indicate the time when at least one piece of media information needs to be presented. Presentation segment information; the presentation segment information is used to indicate at least one portion of the media information that needs to be presented. Presenting transition effect information; the presenting transition effect information is used to indicate the presentation effect of the start or end process of the media information; Present dependency information; The presentation dependency information is used to indicate the second media information that the first media information needs to depend on when it is presented. Present alternative information; The presentation substitution information is used to indicate that when the first media information in the media information cannot be presented, the first media information can be replaced by the second media information.

14. The method according to claim 11, wherein, The presentation control information is encapsulated into the file to be decoded using an ISOBMFF data box.

15. The method according to claim 14, wherein, The presentation control information includes entity groups; the entity groups are used to associate presentation combinations and metadata tracks, the presentation combinations include media information entities of at least one media type that need to be presented within a preset time period, and the media information entities include media tracks or media data items.

16. The method of claim 14, wherein, The presentation control information includes entity groups; the entity groups are used to associate presentation combinations, the presentation combinations include media information entities of at least one media type that need to be presented within a preset time period, the media information entities include media tracks or media data items, and the presentation information of the presentation combination is indicated by the metadata of the presentation combination.

17. The method of claim 14, wherein, The presentation control information includes entity groups; the entity groups are used to associate presentation combinations, the presentation combinations include media information entities of at least one media type that need to be presented within a preset time period, the media information entities include media tracks or media data items, the presentation information of the presentation combination is indicated by the information data box of the presentation combination, and the information data box is contained in a pre-selected data box.

18. A communication device, wherein, include: Memory and processor; Memory and processor are coupled; The memory is used to store instructions that can be executed by the processor; When the processor executes the instructions, it performs the method as described in any one of claims 1 to 17.

19. A computer-readable storage medium, wherein, The computer-readable storage medium includes a non-transitory computer-readable storage medium on which computer instructions are stored, which, when executed on a communication device, cause the communication device to perform the method as described in any one of claims 1-17.

20. A computer program product, wherein, When the computer program product is executed, it implements the method as described in any one of claims 1 to 17.

Images