Multimedia data processing method, device, and computer-readable storage medium
Patent Information
- Application Number
- PCT/CN2025/085665
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-03-28
- Publication Date
- 2026-10-01
Smart Images
Figure CN2025085665_01102026_PF_FP_ABST
Abstract
Description
Multimedia data processing methods, devices and computer-readable storage media Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus and computer-readable storage medium for processing multimedia data. Background Technology
[0002] During real-time playback of multimedia data such as audio and video, packet loss often occurs due to unstable data transmission links between the sending and receiving ends, leading to incomplete playback data. To ensure data integrity during real-time playback, buffering is often used to retransmit lost data and delay playback. This introduces a certain delay in the playback of multimedia data, failing to meet real-time requirements. Particularly in audio and video recording scenarios, such as short film shooting and interview recording, users expect real-time monitoring while ensuring high integrity of the final product. The requirement for data integrity during real-time playback is correspondingly lower. In such scenarios, a processing solution that can guarantee both real-time playback and final product integrity is urgently needed. Summary of the Invention
[0003] Based on this, this application provides a method, apparatus, and computer-readable storage medium for processing multimedia data, so as to ensure the real-time playback of multimedia data while ensuring the integrity of the multimedia data as a whole.
[0004] In a first aspect, this application provides a method for processing multimedia data, including:
[0005] During the acquisition of multimedia data, the first multimedia data stream is played.
[0006] In response to the situation that the first multimedia data stream has incomplete playback data during playback, a second multimedia data stream is generated. The second multimedia data stream contains the data of the first multimedia data stream and the data of the first multimedia data stream that needs to be supplemented and completed.
[0007] Store the second multimedia data stream.
[0008] Secondly, this application also provides a method for processing multimedia data, including:
[0009] During the acquisition of multimedia data, the acquired multimedia data stream is output, which includes all the data of the first multimedia data stream played during the acquisition process.
[0010] In response to the situation that the first multimedia data stream has incomplete playback data during playback, target multimedia data is obtained, and the target multimedia data is used to indicate the data to be supplemented and completed in the first multimedia data stream;
[0011] The target multimedia data is output, wherein the target multimedia data is used to generate a second multimedia data stream, the second multimedia data stream contains the data of the first multimedia data stream and the data to be supplemented, and the second multimedia data stream is used for storage.
[0012] Thirdly, this application also provides a multimedia data processing device, including a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and, when executing the computer program, to implement the above-described multimedia data processing method.
[0013] Fourthly, this application also provides a computer-readable storage medium, the computer-readable storage medium including a stored computer program, wherein the computer program, when run by a processor, controls the device where the storage medium is located to perform the above-described multimedia data processing method.
[0014] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description
[0015] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0016] Figure 1 is a schematic flowchart of the steps of a multimedia data processing method provided in an embodiment of this application;
[0017] Figure 2 is a schematic diagram of the structure of a MIC device provided in an embodiment of this application;
[0018] Figure 3 is a schematic flowchart of another multimedia data processing method provided in an embodiment of this application;
[0019] Figure 4 is a schematic diagram of a user control interface of a data receiving device provided in an embodiment of this application;
[0020] Figure 5 is a schematic diagram of the user control interface of a data transmitting terminal device provided in an embodiment of this application;
[0021] Figure 6 is a schematic diagram of the settings interface of a data receiving device provided in an embodiment of this application;
[0022] Figure 7 is a schematic diagram of the structure of a MIC device provided in an embodiment of this application;
[0023] Figure 8 is a schematic diagram of another MIC device provided in an embodiment of this application;
[0024] Figure 9 is a schematic flowchart of another multimedia data processing method provided in an embodiment of this application;
[0025] Figure 10 is a schematic block diagram of a multimedia data processing device provided in an embodiment of this application. Detailed Implementation
[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0027] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.
[0028] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.
[0029] It should also be understood that the term "and / or" as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0030] This application provides a multimedia data processing method, apparatus, and computer-readable storage medium to ensure both real-time playback and the integrity of multimedia data. The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.
[0031] Please refer to Figure 1, which is a schematic flowchart of a multimedia data processing method provided by an embodiment of this application. This multimedia data processing method can be applied to a data transmitting device, a data receiving device, or a system composed of a data transmitting device and a data receiving device. Some steps are performed by the data transmitting device, and some steps are performed by the data receiving device; this application does not impose any limitations on this method.
[0032] The data transmitting device may include a MIC (Microphone) device, such as a wireless MIC transmitter. The data transmitting device may also include a camera, a webcam, or other similar devices. This application does not limit the scope of the application.
[0033] The data receiving device may include a MIC (Microphone) device, such as a wireless MIC receiver. The data receiving device may also include mobile phones, speakers, headphones, audio players, video players, etc., which are not limited in this application.
[0034] For ease of explanation, the following description of the embodiments mainly uses the MIC device as an example to illustrate the multimedia data processing method. The same applies to other types of data transmitting or receiving devices, and will not be repeated.
[0035] Taking microphone (MIC) devices as an example, in related technologies, most solutions for transmitting audio data recorded by the MIC device to other receiving devices only support real-time transmission (monitoring). During real-time transmission, if the network environment is poor, packet loss may occur. To ensure the integrity of the audio data, packet loss retransmission is often performed, which results in a certain delay in the played audio data, failing to meet real-time requirements. Especially if the MIC device and receiving device wirelessly disconnect for a period of time due to very poor network conditions, this will cause packet loss, preventing real-time transmission and reception by the receiving device, and thus preventing real-time playback of the audio data.
[0036] In reality, in some scenarios, such as live streaming, users have high requirements for real-time playback; while in other scenarios, such as short film shooting and interview recording, users expect the finished product to have a high degree of integrity. Therefore, how to ensure both real-time playback (i.e., real-time monitoring capability) and the integrity of the finished product is an urgent problem to be solved.
[0037] To address the aforementioned issues, this application provides the following method for processing multimedia data.
[0038] As shown in Figure 1, the multimedia data processing method specifically includes steps S101 to S103. In this embodiment, it can be executed by at least one of the data transmitting end device or the data receiving end device.
[0039] S101. During the acquisition of multimedia data, the first multimedia data stream is played.
[0040] The multimedia data mentioned in this application may include at least one of audio data and video data. For example, when multimedia data is collected via a data transmitting device, the first multimedia data stream is generated by the data transmitting device collecting the multimedia data. A data stream refers to multiple data items arranged in chronological order; that is, the first multimedia data stream is a collection of multiple data items arranged in chronological order generated by the data transmitting device collecting multimedia data. For instance, when audio data is collected via a microphone device, the first multimedia data stream is generated by the microphone device collecting the audio data. The first multimedia data stream may include multiple audio data items arranged in chronological order.
[0041] During the acquisition of multimedia data, the first multimedia data stream can be played by the data receiving device. Alternatively, the first multimedia data stream can also be played by the data transmitting device.
[0042] Since the first multimedia data stream is played during the multimedia data acquisition process, it can play even when the playback data in the first multimedia data stream is incomplete. Compared with the method of playing after packet loss and retransmission, this ensures the real-time performance of data playback.
[0043] It is understood that the first multimedia data stream can be played in real time during the transmission of multimedia data. That is, in some embodiments, the playback process of the first multimedia data stream can also be synchronized in real time with the transmission process of the first multimedia data stream. In some embodiments, the playback process of the first multimedia data stream can also refer to the transmission process of the first multimedia data stream.
[0044] In some embodiments, playing the obtained first multimedia data stream includes: playing the first multimedia data stream during the process of receiving the first multimedia data stream sent by the data transmitting device.
[0045] For example, taking a microphone (MIC) device as an example, as shown in Figure 2, the MIC device 1000 includes a transmitter (TX) 100 and a receiver (RX) 200. The transmitter 100 and receiver 200 are wirelessly connected. The transmitter 100 collects audio data and sends the collected audio data stream to the receiver 200. The receiver 200 receives the audio data stream sent by the transmitter 100 and plays the audio data stream during the audio data collection process at the transmitter 100. The audio data stream can be played even when the first multimedia data stream has incomplete playback data, thereby reducing playback latency caused by retransmission due to incomplete audio data and ensuring real-time performance.
[0046] In other embodiments, the data receiving device receives the first multimedia data stream sent by the data transmitting device, caches the first multimedia data stream locally on the data receiving device, and plays the locally cached first multimedia data. Although there will be a certain delay, from the user's perspective, it will not affect the user's real-time listening experience. By caching the first multimedia data stream for playback, the stability and reliability of the playback can be guaranteed.
[0047] For example, playing the obtained first multimedia data stream includes: playing the audio data of the first multimedia data stream using an audio output device; and / or playing the video data of the first multimedia data stream using a display device.
[0048] For example, the first multimedia data stream includes audio data, which is played through an audio output device, including but not limited to speakers, headphones, and loudspeakers, etc., which are not limited in this application.
[0049] For another example, the first multimedia data stream includes video data, which is played through a display device. The display device includes, but is not limited to, LCD (Liquid Crystal Display), LED (Light-Emitting Diode), OLED (Organic Light-Emitting Diode) displays, etc., and is not limited in this application.
[0050] S102. In response to the situation that the playback data of the first multimedia data stream is incomplete during playback, a second multimedia data stream is generated. The second multimedia data stream contains the data of the first multimedia data stream and the data of the first multimedia data stream that needs to be supplemented and completed.
[0051] During the playback of the first multimedia data stream, due to factors such as network conditions, the playback data of the first multimedia data stream may be incomplete. To ensure the integrity of the final product, a second multimedia data stream with complete data is generated. For example, the second multimedia data stream includes an audio data stream and / or a video data stream.
[0052] Incomplete playback data can refer to frame drops during playback, such as audio data interruption or stuttering, or video data momentarily going black or flickering.
[0053] It is understood that the process of generating the second multimedia data stream can be executed when incomplete playback data is detected in the first multimedia data stream during playback. This generation process can overlap with the playback process in time, or it can be executed after playback has ended. The embodiments of this application are not limited to immediately generating the second multimedia data stream upon detecting incomplete playback data; the prerequisite for generating the second multimedia data stream can be the existence of incomplete playback data.
[0054] It is understandable that the data to be supplemented in the first multimedia data stream can be data lost during the transmission of the first multimedia data stream from the data transmitting device to the data receiving device, such as packet loss. In this way, supplementing the data lost during the transmission of the first multimedia data stream can improve the data integrity of the final stored second multimedia data stream and enhance the overall quality of the final product.
[0055] In some embodiments, the process of acquiring multimedia data in the above steps can be the process of acquiring the complete data stream corresponding to the first multimedia data stream.
[0056] The second multimedia data stream contains not only the data from the first multimedia data stream but also the data from the first multimedia data stream that needs to be supplemented and completed. In some embodiments, the first multimedia data stream can be cleared from the data receiving device after playback is complete by clearing the cache or direct deletion, while the second multimedia data stream can be converted to non-transient storage instead of caching. In some embodiments, the first multimedia data stream can be used only for playback and / or caching during multimedia data acquisition, while the second multimedia data stream can be used only for non-transient storage during multimedia data acquisition. The data from the first multimedia data stream and the data to be supplemented and completed in the second multimedia data stream can be arranged in chronological order.
[0057] The second multimedia data stream can be generated by either the data receiving device or the data transmitting device; this application does not impose any limitation on this.
[0058] S103, Store the second multimedia data stream.
[0059] After generating the second multimedia data stream, the second multimedia data stream is stored in a corresponding storage device. For example, after the data receiving device generates the second multimedia data stream, it stores the second multimedia data stream in the storage device of the data receiving device, wherein the storage device includes, but is not limited to, internal memory, external memory, etc.
[0060] By storing the second multimedia data stream, users can obtain the complete video data simply by retrieving the stored second multimedia data stream. In this application, by playing the first multimedia data stream during multimedia data acquisition and supplementing the data of the first multimedia data stream to form a complete second multimedia data stream for storage when the first multimedia data playback is incomplete, both real-time playback and complete video data are guaranteed.
[0061] In some embodiments, as shown in FIG3, before step S102 generates the second multimedia data stream, the multimedia data processing method further includes step S104, and step S102 includes sub-step S1021.
[0062] S104. In response to the situation that the playback data of the first multimedia data stream is incomplete during playback, target multimedia data is obtained, and the target multimedia data is used to indicate the data to be supplemented and completed in the first multimedia data stream.
[0063] S1021. Generate a second multimedia data stream based on the first multimedia data stream and the target multimedia data.
[0064] The target multimedia data is the data to be supplemented and completed during the playback of the first multimedia data stream. For example, the target multimedia data is obtained during the playback of the first multimedia data stream, that is, the operation of obtaining the data to be supplemented and completed is performed during the playback of the first multimedia data stream.
[0065] Alternatively, the target multimedia data can be obtained after the first multimedia data stream has finished playing. That is, the operation to obtain the data to be completed is not performed during the playback of the first multimedia data stream, but rather after its completion. This allows the data to be completed during idle periods, reducing the bandwidth consumed by real-time data transmission during playback and ensuring the real-time performance of the streaming.
[0066] For example, the target multimedia data includes multiple data points to be supplemented from a first multimedia data stream. For instance, the target multimedia data includes multiple data packets, each containing data points to be supplemented from the first multimedia data stream of a corresponding size. These multiple data points to be supplemented are different data points corresponding to different times in the first multimedia data stream. For example, each data point to be supplemented corresponds to a unique timestamp, and each data point to be supplemented corresponds to a different timestamp.
[0067] For example, multiple pieces of data to be supplemented are obtained all at once after the first multimedia data stream has finished playing. That is, during the playback of the first multimedia data stream, the operation of obtaining the data to be supplemented is not performed, but rather multiple pieces of data to be supplemented are obtained all at once after the first multimedia data stream has finished playing. Alternatively, multiple pieces of data to be supplemented are obtained multiple times during the playback of the first multimedia data stream. That is, during the playback of the first multimedia data stream, the operation of obtaining the data to be supplemented is performed multiple times, and the data to be supplemented is different each time. The target multimedia data is obtained based on the multiple pieces of data to be supplemented. This application does not limit the method of obtaining the target multimedia data.
[0068] In some embodiments, obtaining target multimedia data includes: obtaining target multimedia data in response to a triggering condition being met.
[0069] In other words, the operation of obtaining the target multimedia data will only be performed if the triggering condition is met. It can be understood that the triggering condition can be a condition during the playback of the first multimedia data stream, or a condition after the first multimedia data stream has finished playing.
[0070] For example, the triggering conditions include at least one of user-instructed conditions and preset conditions. User-instructed conditions represent conditions that can be met through user-instructed actions, while preset conditions represent conditions that can be met automatically without user-instructed actions and are unrelated to user-instructed actions. That is, the acquisition of target multimedia data can be actively triggered by user-instructed actions or can be automatically triggered; this application does not impose any limitations.
[0071] For example, satisfying user-instructed conditions includes:
[0072] The user instructs the data receiving device to pause or terminate the recording of the first multimedia data stream;
[0073] The user instructs the data transmitting device to pause or terminate the acquisition of the first multimedia data stream;
[0074] The user instructs the data receiving device or data transmitting device to activate a preset working mode. The preset working mode is used to indicate the acquisition of target multimedia data when there is incomplete playback data during the first multimedia data stream playback. The first multimedia data stream is transmitted from the data transmitting device to the data receiving device.
[0075] It should be noted that the acquisition of the first multimedia data stream can refer to the data acquisition operation of the first multimedia data stream. The recording of the first multimedia data stream can refer to the data storage operation of the first multimedia data stream. When the data receiving device pauses or terminates the recording of the first multimedia data stream, the acquisition of the first multimedia data stream by the data transmitting device may continue and is not necessarily paused or terminated.
[0076] For example, the data receiving device is equipped with a pause or stop recording control. This pause or stop recording control includes, but is not limited to, physical buttons and user interface controls. It responds to the user's instruction to pause or stop recording the first multimedia data stream, thereby obtaining the target multimedia data. For instance, as shown in Figure 4, a "Stop Recording" control is displayed on the user interface of the data receiving device. If the user clicks this "Stop Recording" control, the target multimedia data is obtained. Since the user instructs to pause or stop recording the first multimedia data stream, there is no experience of disconnected listening for the user, ensuring the real-time playback of the first multimedia data stream.
[0077] For another example, the data transmitting device is equipped with a pause or stop acquisition control. This pause or stop acquisition control includes, but is not limited to, physical buttons and user interface controls. It responds to the user's instruction to pause or stop the acquisition of the first multimedia data stream, thereby obtaining the target multimedia data. For instance, as shown in Figure 5, a "Pause Acquisition" control is displayed on the user interface of the data transmitting device. If the user clicks this "Pause Acquisition" control, the target multimedia data is obtained. Since the user instructs to pause or stop the acquisition of the first multimedia data stream, there is no experience issue of disconnected listening for the user, ensuring the real-time playback of the first multimedia data stream.
[0078] For another example, the data receiving or transmitting device is configured with a preset working mode and a corresponding switch control. In this preset working mode, the data receiving or transmitting device automatically performs data completion processing. The switch control for the preset working mode includes, but is not limited to, physical buttons and user interface controls. The device responds to the user's instruction to activate the preset working mode using the switch control, thereby obtaining the target multimedia data. For example, as shown in Figure 6, the settings interface of the data receiving device displays a preset working mode switch control such as "File Repair Mode." If the user activates this switch control, the data receiving device activates the preset working mode and obtains the target multimedia data. Since the preset working mode is activated by the user's instruction, there is no experience issue of disconnected listening for the user, ensuring the real-time playback of the first multimedia data stream.
[0079] For example, satisfying the preset conditions includes:
[0080] During the process of transmitting the first multimedia data stream from the data transmitting device to the data receiving device, there is additional data transmission bandwidth between the data receiving device and the data transmitting device.
[0081] For example, suppose the transmission bandwidth between the data receiving device and the data transmitting device is a MHz, while the data transmission bandwidth occupied by the data transmitting device to transmit the first multimedia data stream to the data receiving device is b MHz, where b < a. That is, the data receiving device and the data transmitting device have additional data transmission bandwidth. In this case, the target multimedia data is obtained. Because the data receiving device and the data transmitting device have additional data transmission bandwidth, the operation of obtaining the target multimedia data does not affect the transmission of the first multimedia data stream, does not cause delay, and ensures the real-time playback of the first multimedia data stream.
[0082] Alternatively, the target multimedia data can be acquired during the idle window corresponding to the idle time period of both the data receiving and transmitting devices. Since the target multimedia data is acquired based on the idle window, the operation of acquiring the target multimedia data will not affect the transmission of the first multimedia data stream from the data transmitting device to the data receiving device, and will not cause delay. Therefore, the real-time playback of the first multimedia data stream is guaranteed.
[0083] In some embodiments, obtaining target multimedia data includes: receiving target multimedia data sent by a data transmitting device, wherein the target multimedia data is generated by the data transmitting device.
[0084] Since the playback data of the first multimedia data stream may be incomplete due to loss during data transmission from the data transmitting device to the data receiving device, the data transmitting device can store or cache data from the first multimedia data stream that needs to be supplemented to complete the stream. It can be understood that the data stream acquired by the data transmitting device can be a complete data stream, containing all the data from the second data stream. Specifically, the transmitting device can store or cache the complete data stream during the multimedia data acquisition process.
[0085] In response to the situation where the playback data of the first multimedia data stream is incomplete during playback, the data transmitting device generates target multimedia data and sends the generated target multimedia data to the data receiving device. The data receiving device receives the target multimedia data sent by the data transmitting device, thereby completing the data of the first multimedia data stream.
[0086] For example, in response to the fulfillment of a triggering condition, the data transmitting device sends the target multimedia data to the data receiving device, and the data receiving device receives the target multimedia data sent by the data transmitting device. The fulfillment of the triggering condition is as described in the embodiments above, and therefore will not be repeated here.
[0087] In some embodiments, before receiving target multimedia data sent by a data transmitting device, the method further includes: sending data information to be supplemented and completed corresponding to a first multimedia data stream to the data transmitting device, so that the data transmitting device can generate target multimedia data based on the data information to be supplemented and completed.
[0088] A data transmitting device transmits a first multimedia data stream to a data receiving device. In response to data loss, the data receiving device sends data information corresponding to the first multimedia data stream that needs to be supplemented back to the data transmitting device. For example, the data information to be supplemented includes a data identifier, which identifies the time information of the data to be supplemented within the first multimedia data stream. The time information includes, but is not limited to, time nodes (such as timestamps) and sequence numbers arranged chronologically. The data transmitting device generates target multimedia data based on the received data information to be supplemented.
[0089] In other embodiments, the data transmitting device includes a controller, and the data information to be supplemented is obtained by the data transmitting device receiving information reported by the controller. The data transmitting device generates target multimedia data based on the obtained data information to be supplemented.
[0090] For example, taking a microphone (MIC) device as an example, as shown in Figure 7, the MIC device 1000 includes a transmitter (TX) 100 and a receiver (RX) 200. The transmitter 100 includes a first radio frequency (RF) module 110, and the receiver 200 includes a second radio frequency (RF) module 210. The transmitter 100 transmits a first multimedia data stream based on the first RF module 110, and the receiver 200 receives the first multimedia data stream based on the second RF module 210. The first RF module 110 includes a controller 111. During the process of the transmitter 100 transmitting the first multimedia data stream to the receiver 200 based on the first RF module 110, the first RF module 110 receives reporting information from the controller 111. Based on the reporting information, it obtains the data information to be supplemented and completed corresponding to the first multimedia data stream. For example, the controller 111 reports to the first RF module 110 which data packet of the first multimedia data stream failed to be transmitted. The transmitter 100 generates target multimedia data based on the obtained data information to be supplemented and completed.
[0091] In other embodiments, the data transmitting device includes a buffer, in which the first multimedia data stream is cached. The data information to be supplemented is obtained by the data transmitting device monitoring the overflow of the buffer. For example, taking the MIC device 1000 shown in FIG. 7 as an example, the transmitting end 100 also includes a buffer (not shown in the figure). The first radio frequency module 110 receives the overflow of the buffer and obtains the data information to be supplemented corresponding to the first multimedia data stream based on the overflow of the buffer. For example, it obtains which data packet of the first multimedia data stream cached in the buffer overflowed. The transmitting end 100 generates target multimedia data according to the obtained data information to be supplemented.
[0092] It should be noted that the data transmitting device may generate target multimedia data based on the data information to be supplemented in a manner not limited to the methods listed above, and no limitation is made in this application.
[0093] For example, after generating target multimedia data, the data transmitting device stores the target multimedia data. For instance, the target multimedia data is stored in a storage device, which includes at least one of the data transmitting device's internal memory and external memory. When a trigger condition is met, the data transmitting device sends the stored target multimedia data to the data receiving device. For example, still taking the MIC device 1000 shown in FIG. 7 as an example, the transmitting end 100 also includes a first memory card (SD card) 120. After generating the target multimedia data, the transmitting end 100 stores the target multimedia data in the first memory card 120. In response to the trigger condition being met, the transmitting end 100 sends the target multimedia data stored in the first memory card 120 to the receiving end 200.
[0094] By storing the generated target multimedia data, the stored target multimedia data is only sent to the data receiving device when the triggering condition is met. This does not affect the data transmitting device from sending the first multimedia data stream to the data receiving device, thus ensuring the real-time playback of the first multimedia data stream.
[0095] For example, after generating target multimedia data, the data transmitting device encodes the target multimedia data and sends the encoded target multimedia data to the data receiving device. For instance, taking the MIC device 1000 shown in Figure 7 as an example, the transmitting end 100 also includes an encoder 130, and the receiving end 200 includes a decoder 220. The transmitting end 100 encodes the generated target multimedia data using the encoder 130 and transmits it as a bitstream. The transmitting end 100 sends the encoded target multimedia data to the receiving end 200 based on the first radio frequency module 110. After receiving the encoded target multimedia data based on the second radio frequency module 210, the receiving end 200 decodes the encoded target multimedia data using the decoder 220.
[0096] Encoding the target multimedia data and then transmitting the encoded target multimedia data improves the reliability and efficiency of target multimedia data transmission.
[0097] For example, after generating the target multimedia data, the data transmitting device outputs the target multimedia data in sequence based on the data identifier corresponding to the target multimedia data, and sends the target multimedia data to the data receiving device. The data identifier is used to identify the time information of the target multimedia data, including but not limited to time nodes (such as timestamps) and sequence numbers arranged chronologically. By outputting the target multimedia data in sequence, the reliability of the target multimedia data transmission can be improved.
[0098] For example, during the playback of the first multimedia data stream, after the data transmitting device generates the target multimedia data, it outputs the target amount of target multimedia data each time and sends the target amount of target multimedia data to the data receiving device. The target data amount can be determined by the additional data transmission bandwidth available to both the data receiving device and the data transmitting device. By controlling the amount of target multimedia data output each time, the transmission of the first multimedia data stream from the data transmitting device to the data receiving device is not affected, thus ensuring the real-time playback of the first multimedia data stream.
[0099] In some embodiments, the method further includes: acquiring a third multimedia data stream, wherein the third multimedia data stream and the first multimedia data stream are acquired by different devices; obtaining target multimedia data, including: obtaining target multimedia data based on the data of the third multimedia data stream; wherein the third multimedia data stream is acquired synchronously during the acquisition of the first multimedia data stream.
[0100] For example, a data transmitting device acquires a first multimedia data stream and sends it to a data receiving device. During the acquisition of the first multimedia data stream, the data receiving device simultaneously acquires a third multimedia data stream. In response to incomplete data in the first multimedia data stream, the data receiving device obtains the target multimedia data based on the data from the synchronously acquired third multimedia data stream. Since the target multimedia data is obtained through the synchronously acquired third multimedia data stream by the data receiving device, compared to the data transmitting device generating the target multimedia data and sending it to the data receiving device, the process of the data transmitting device sending the target multimedia data to the data receiving device is eliminated, thus improving the efficiency of obtaining the target multimedia data.
[0101] For example, in some scenarios, a data receiving device can possess both data receiving and data acquisition capabilities. For instance, in addition to a communication module, the data receiving device can include an additional audio acquisition module, such as a microphone module, for acquiring audio data. Thus, while the data sending device is acquiring audio data through its own audio acquisition module, the data receiving device is simultaneously acquiring audio data. Therefore, in scenarios where the data receiving and sending ends are relatively close, the simultaneous acquisition of audio data by the data receiving device can be used to complete the first multimedia data stream.
[0102] In some embodiments, obtaining target multimedia data based on data from a third multimedia data stream includes: obtaining target multimedia data from the third multimedia data stream based on data information to be supplemented and completed corresponding to a first multimedia data stream.
[0103] For example, during the acquisition of the first multimedia data stream, the data receiving device acquires the data information to be supplemented corresponding to the first multimedia data stream and simultaneously collects the third multimedia data stream. Based on the data information to be supplemented corresponding to the first multimedia data stream, the target multimedia data is obtained from the third multimedia data stream simultaneously collected by the data receiving device. The data receiving device does not need to obtain the target multimedia data from the data transmitting device, thus improving the efficiency of obtaining the target multimedia data.
[0104] It is understood that a multimedia data stream can simultaneously contain audio and video data. Therefore, in some scenarios, a second multimedia data stream can contain both audio and video data. For example, the data sending device may include an audio acquisition module, such as a microphone module, while the data receiving device may include a video acquisition module, such as a camera module. In this case, the first multimedia data stream can be an audio data stream, and the data receiving device can simultaneously acquire a video data stream while receiving the audio data stream. The completed audio data stream and the acquired video data stream can then be combined to form the second multimedia data stream, ensuring the integrity of the audio and video in the final product. Of course, in other embodiments, the data sending device may also include a video acquisition module, and the first multimedia data stream may be a video data stream or an audio / video data stream; this is not limited here.
[0105] In some embodiments, the method further includes: obtaining a fourth multimedia data stream during the process of obtaining the first multimedia data stream, wherein the fourth multimedia data stream and the first multimedia data stream respectively include an audio data stream and a video data stream; generating a second multimedia data stream, including: generating the second multimedia data stream based on the target multimedia data, the first multimedia data stream, and the fourth multimedia data stream.
[0106] The fourth multimedia data stream can be collected by a data receiving device, a data transmitting device, or a terminal other than the data receiving and transmitting devices. The fourth multimedia data stream and the first multimedia data stream each include either audio or video data streams; that is, the data in the fourth multimedia data stream and the data in the first multimedia data stream are of different data types. For example, the first multimedia data stream includes audio data streams, and the fourth multimedia data stream includes video data streams. Or, the first multimedia data stream includes video data streams, and the fourth multimedia data stream includes audio data streams.
[0107] The second multimedia data stream can be generated by the data receiving device based on the target multimedia data, the first multimedia data stream, and the fourth multimedia data stream. Alternatively, the second multimedia data stream can also be generated by the data transmitting device based on the target multimedia data, the first multimedia data stream, and the fourth multimedia data stream. For example, as shown in Figure 8, the receiving end 200 of the MIC device 1000 has a built-in video recording device 230. It acquires audio data through the transmitting end 100 to obtain the first multimedia data stream, which is then played and output by the receiving end 200. Simultaneously, it acquires video data through the video recording device 230 to obtain the fourth multimedia data stream. In response to incomplete playback data in the first multimedia data stream, the transmitting end 100 generates target multimedia data and sends it to the receiving end 200. The receiving end 200 generates the second multimedia data stream based on the target multimedia data, the first multimedia data stream, and the fourth multimedia data stream. The second multimedia data stream includes both audio and video data.
[0108] For example, after generating a second multimedia data stream based on the target multimedia data, the first multimedia data stream, and the fourth multimedia data stream, the second multimedia data stream is stored. For instance, as shown in FIG8, the receiving end 200 also includes a second memory card (SD card) 240, which stores the second multimedia data stream in the second memory card 240 after generating the second multimedia data stream based on the target multimedia data, the first multimedia data stream, and the fourth multimedia data stream.
[0109] For example, generating a second multimedia data stream based on target multimedia data, a first multimedia data stream, and a fourth multimedia data stream includes: completing the data of the first multimedia data stream based on the target multimedia data; and combining the fourth multimedia data stream with the completed first multimedia data stream to generate the second multimedia data stream.
[0110] For example, based on the time information corresponding to the target multimedia data and the first multimedia data stream, where the time information includes, but is not limited to, time nodes (such as timestamps) and sequence numbers arranged chronologically, the target multimedia data is used to complete the data of the first multimedia data stream. In the completed first multimedia data stream, the target multimedia data and the data of the first multimedia data stream are arranged in chronological order. Then, the fourth multimedia data stream and the completed first multimedia data stream are combined to generate a second multimedia data stream. For example, based on time information such as timestamps, the fourth multimedia data stream and the completed first multimedia data stream are combined to obtain a video file containing a complete audio track.
[0111] Please refer to Figure 9, which is a schematic flowchart of another multimedia data processing method provided by an embodiment of this application. As shown in Figure 9, the multimedia data processing method specifically includes steps S201 to S203.
[0112] S201. During the acquisition of multimedia data, the acquired multimedia data stream is output. The multimedia data stream includes all the data of the first multimedia data stream that is played during the acquisition process.
[0113] S202. In response to the situation that the playback data of the first multimedia data stream is incomplete during playback, target multimedia data is obtained, which is used to indicate the data to be supplemented and completed in the first multimedia data stream.
[0114] S203. Output target multimedia data, wherein the target multimedia data is used to generate a second multimedia data stream, the second multimedia data stream contains the data of the first multimedia data stream and the data to be supplemented and completed, and the second multimedia data stream is used for storage.
[0115] It is understandable that the multimedia data stream obtained in S201 can be the complete data stream corresponding to the first multimedia data stream mentioned above.
[0116] For example, multimedia data is acquired by a data transmitting device to obtain a multimedia data stream, and then the acquired multimedia data stream is output. During the acquisition of multimedia data, a data receiving device can play the first multimedia data stream included in the multimedia data stream. Of course, the data transmitting device can also play the first multimedia data stream.
[0117] This embodiment can be executed by a multimedia data transmitting device. Further descriptions and limitations of the relevant features in this embodiment can be found in the relevant sections of the foregoing embodiments, and will not be repeated here.
[0118] Please refer to Figure 10, which is a schematic block diagram of a multimedia data processing device provided in an embodiment of this application.
[0119] As shown in Figure 10, the multimedia data processing device 300 may include at least one processor 310 and at least one memory 320, which are connected by a bus, such as an I2C (Inter-integrated Circuit) bus.
[0120] Specifically, the processor 310 can be a microcontroller unit (MCU), a central processing unit (CPU), or a digital signal processor (DSP), etc.
[0121] Specifically, the memory 320 can be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a portable hard drive, etc. The memory 320 stores various computer programs for the processor 310 to execute.
[0122] The processor 310 is used to execute a computer program and, when executing the computer program, implement the method as described in the above embodiments.
[0123] For example, the multimedia data processing device 300 includes a MIC device, which may be the MIC device 1000 in the foregoing embodiments.
[0124] The relevant descriptions and implementation methods in the embodiments of this application can be found in the relevant introductions in the foregoing method embodiments, and will not be repeated here.
[0125] The embodiments of this application also provide a computer-readable storage medium, which includes a stored computer program that, when run by a processor, controls the device where the computer-readable storage medium is located to execute the steps of the multimedia data processing method provided in the embodiments of this application.
[0126] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., provided on the computer device.
[0127] The relevant descriptions and implementation methods in the embodiments of this application can be found in the relevant introductions in the foregoing method embodiments, and will not be repeated here.
[0128] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A method for processing multimedia data, characterized in that, include: During the acquisition of multimedia data, the first multimedia data stream is played. In response to the situation where the first multimedia data stream has incomplete playback data during playback, a second multimedia data stream is generated, which includes the data from the first multimedia data stream and the data to be supplemented in the first multimedia data stream. Store the second multimedia data stream.
2. The method according to claim 1, characterized in that, Before generating the second multimedia data stream, the method further includes: In response to the situation that the first multimedia data stream has incomplete playback data during playback, target multimedia data is obtained, and the target multimedia data is used to indicate the data to be supplemented and completed in the first multimedia data stream; The generation of the second multimedia data stream includes: The second multimedia data stream is generated based on the first multimedia data stream and the target multimedia data.
3. The method according to claim 2, characterized in that, The target multimedia data includes multiple data points from the first multimedia data stream that need to be supplemented and completed.
4. The method according to claim 3, characterized in that, The multiple data to be supplemented are different data corresponding to different times in the first multimedia data stream.
5. The method according to claim 4, characterized in that, The multiple data points to be supplemented are obtained all at once after the first multimedia data stream has finished playing, or the multiple data points to be supplemented are obtained multiple times during the playback of the first multimedia data stream.
6. The method according to claim 2, characterized in that, The target multimedia data is obtained during the playback of the first multimedia data stream, or the target multimedia data is obtained after the playback of the first multimedia data stream has ended.
7. The method according to claim 2, characterized in that, The acquisition of target multimedia data includes: In response to the fulfillment of the triggering condition, the target multimedia data is obtained.
8. The method according to claim 7, characterized in that, The triggering conditions include at least one of user-instructed conditions and preset conditions.
9. The method according to claim 8, characterized in that, The conditions for satisfying the user instruction include: The user instructs the data receiving device to pause or terminate the recording of the first multimedia data stream; The user instructs the data transmitting device to pause or terminate the acquisition of the first multimedia data stream; The user instructs the data receiving device or data transmitting device to activate a preset working mode, which is used to indicate that the target multimedia data is obtained when there is incomplete playback data during the playback of the first multimedia data stream; wherein, the first multimedia data stream is transmitted from the data transmitting device to the data receiving device.
10. The method according to claim 8, characterized in that, The preset conditions include: During the process of transmitting the first multimedia data stream from the data transmitting device to the data receiving device, the data receiving device and the data transmitting device have additional data transmission bandwidth.
11. The method according to claim 1, characterized in that, The first multimedia data stream is generated by the data transmitting device collecting the multimedia data.
12. The method according to claim 11, characterized in that, Playing the obtained first multimedia data stream includes: During the process of receiving the first multimedia data stream sent by the data transmitting device, the first multimedia data stream is played.
13. The method according to claim 2, characterized in that, The acquisition of target multimedia data includes: The target multimedia data is received from the data transmitting device, wherein the target multimedia data is generated by the data transmitting device.
14. The method according to claim 13, characterized in that, Before receiving the target multimedia data sent by the data transmitting device, the method further includes: The data information to be supplemented and completed corresponding to the first multimedia data stream is sent to the data transmitting device, so that the data transmitting device can generate the target multimedia data based on the data information to be supplemented and completed.
15. The method according to claim 14, characterized in that, The data information to be supplemented includes a data identifier, which is used to identify the time information of the data to be supplemented in the first multimedia data stream.
16. The method according to claim 2, characterized in that, The method further includes: A third multimedia data stream is acquired, and the third multimedia data stream and the first multimedia data stream are acquired by different devices; The acquisition of target multimedia data includes: The target multimedia data is obtained based on the data from the third multimedia data stream; the third multimedia data stream is collected synchronously during the process of obtaining the first multimedia data stream.
17. The method according to claim 16, characterized in that, Obtaining the target multimedia data based on the data from the third multimedia data stream includes: Based on the data information to be supplemented and completed corresponding to the first multimedia data stream, the target multimedia data is obtained from the third multimedia data stream.
18. The method according to claim 1, characterized in that, Playing the obtained first multimedia data stream includes: Play the audio data of the first multimedia data stream using an audio output device; and / or, The video data of the first multimedia data stream is played using a display device.
19. The method according to claim 1, characterized in that, The second multimedia data stream includes an audio data stream and / or a video data stream.
20. The method according to claim 2, characterized in that, The method further includes: In the process of obtaining the first multimedia data stream, a fourth multimedia data stream is obtained, wherein the fourth multimedia data stream and the first multimedia data stream respectively include one of an audio data stream and a video data stream; The generation of the second multimedia data stream includes: The second multimedia data stream is generated based on the target multimedia data, the first multimedia data stream, and the fourth multimedia data stream.
21. The method according to claim 20, characterized in that, The step of generating the second multimedia data stream based on the target multimedia data, the first multimedia data stream, and the fourth multimedia data stream includes: The data of the first multimedia data stream is completed based on the target multimedia data; The fourth multimedia data stream is combined with the completed first multimedia data stream to generate the second multimedia data stream.
22. A method for processing multimedia data, characterized in that, include: During the acquisition of multimedia data, the acquired multimedia data stream is output, which includes all the data of the first multimedia data stream played during the acquisition process. In response to the situation that the first multimedia data stream has incomplete playback data during playback, target multimedia data is obtained, and the target multimedia data is used to indicate the data to be supplemented and completed in the first multimedia data stream; The target multimedia data is output, wherein the target multimedia data is used to generate a second multimedia data stream, the second multimedia data stream contains the data of the first multimedia data stream and the data to be supplemented, and the second multimedia data stream is used for storage.
23. The method according to claim 22, characterized in that, The target multimedia data includes multiple data points from the first multimedia data stream that need to be supplemented and completed.
24. The method according to claim 23, characterized in that, The multiple data to be supplemented are different data corresponding to different times in the first multimedia data stream.
25. The method according to claim 24, characterized in that, The multiple data points to be supplemented are obtained all at once after the first multimedia data stream has finished playing, or the multiple data points to be supplemented are obtained multiple times during the playback of the first multimedia data stream.
26. The method according to claim 22, characterized in that, The target multimedia data is obtained during the playback of the first multimedia data stream, or the target multimedia data is obtained after the playback of the first multimedia data stream has ended.
27. The method according to claim 22, characterized in that, The output of the target multimedia data includes: In response to the fulfillment of the triggering condition, the target multimedia data is output.
28. The method according to claim 27, characterized in that, The triggering conditions include at least one of user-instructed conditions and preset conditions.
29. The method according to claim 28, characterized in that, The conditions for satisfying the user instruction include: The user instructs the data receiving device to pause or terminate the recording of the first multimedia data stream; The user instructs the data transmitting device to pause or terminate the acquisition of the first multimedia data stream; The user instructs the data receiving device or data transmitting device to activate a preset working mode, which is used to indicate that the target multimedia data is obtained when there is incomplete playback data during the playback of the first multimedia data stream; wherein, the first multimedia data stream is transmitted from the data transmitting device to the data receiving device.
30. The method according to claim 28, characterized in that, The preset conditions include: During the process of transmitting the first multimedia data stream from the data transmitting device to the data receiving device, the data receiving device and the data transmitting device have additional data transmission bandwidth.
31. The method according to claim 22, characterized in that, The first multimedia data stream is generated by the data transmitting device collecting the multimedia data.
32. The method according to claim 22, characterized in that, The first multimedia data stream is played by the data receiving device during the process of receiving the first multimedia data stream sent by the data transmitting device.
33. The method according to claim 22, characterized in that, The target multimedia data is generated by the data transmitting device.
34. The method according to claim 22, characterized in that, The acquisition of target multimedia data includes: The target multimedia data is generated based on the data information to be supplemented and completed corresponding to the first multimedia data stream.
35. The method according to claim 34, characterized in that, The data information to be supplemented includes a data identifier, which is used to identify the time information of the data to be supplemented in the first multimedia data stream.
36. The method according to claim 34, characterized in that, The data information to be supplemented and completed is obtained by the data transmitting device.
37. The method according to claim 36, characterized in that, The data transmitting device includes a controller, and the data information to be supplemented is obtained by the data transmitting device receiving the information reported by the controller.
38. The method according to claim 36, characterized in that, The data transmitting device includes a buffer, the first multimedia data stream is cached in the buffer, and the data information to be supplemented is obtained by the data transmitting device monitoring the overflow of the buffer.
39. The method according to claim 34, characterized in that, The data information to be supplemented and completed is fed back from the data receiving device to the data transmitting device based on the first multimedia data stream received.
40. The method according to claim 22, characterized in that, After obtaining the target multimedia data, the method further includes: The target multimedia data is stored.
41. The method according to claim 40, characterized in that, The target multimedia data is stored in a storage device, which includes at least one of the internal memory and external memory of the data transmitting device, and the target multimedia data is generated by the data transmitting device.
42. The method according to claim 22, characterized in that, The output of the target multimedia data includes: Based on the data identifier corresponding to the target multimedia data, the target multimedia data is output sequentially, and the data identifier is used to identify the time information of the target multimedia data.
43. The method according to claim 22, characterized in that, The output of the target multimedia data includes: During the playback of the first multimedia data stream, the target multimedia data of the target data volume is output each time, wherein the target data volume is determined by the additional data transmission bandwidth of the data receiving end device and the data transmitting end device.
44. The method according to claim 22, characterized in that, The second multimedia data stream includes an audio data stream and / or a video data stream.
45. A multimedia data processing device, characterized in that, It includes a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and, when executing the computer program, to implement the method of any one of claims 1 to 44.
46. The device according to claim 45, characterized in that, The device includes a microphone (MIC).
47. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed by a processor, it controls the device on which the computer-readable storage medium is located to perform the method of any one of claims 1 to 44.