Data transmission method, apparatus, and electronic device

Abstract

The present application relates to the field of wireless communications. Provided in the embodiments of the present application are a data transmission method, an apparatus, and an electronic device. By means of the characteristic that a NearLink air interface clock in a NearLink technology is synchronized with high precision between a transmitting end device and a receiving end device, the transmitting end device corrects timestamps in a media stream on the basis of the NearLink air interface clock. In this way, timestamps in media streams are based on a unified time reference, thereby helping to improve synchronization efficiency. The method comprises the following steps: acquiring first media data, the first media data comprising a plurality of media streams and a first timestamp of each media stream, the first timestamps being used for synchronizing the plurality of media streams; correcting the first timestamp of each media stream on the basis of the NearLink air interface clock to obtain second media data, the NearLink air interface clock being a clock used for time synchronization in the NearLink technology; and transmitting the second media data.

Description

A data transmission method, apparatus and electronic device Technical Field

[0001] This application relates to the field of wireless communication, and more particularly to a data transmission method, apparatus, and electronic device. Background Technology

[0002] Media data includes time-series media data and non-time-series media data. Time-series media data refers to media data that needs to be processed, transmitted, and played in chronological order, such as video and audio. Non-time-series media data refers to media data that does not depend on chronological order during processing, transmission, and playback, such as text and images. When transmitting time-series media data, the sending device assigns different timestamps to each type of media stream based on its characteristics (e.g., video stream timestamps are based on the video frame rate, and audio stream timestamps are based on the audio sampling rate). When the receiving device receives this time-series media data, it can synchronize the timestamps of different timestamps to achieve synchronization of the various media stream types. However, this synchronization process is relatively inefficient. Summary of the Invention

[0003] This application provides a data transmission method, apparatus, and electronic device that utilizes the high-precision synchronization feature of the star-flash air interface clock in star-flash technology between the transmitting and receiving devices. The transmitting device corrects the timestamps in the media stream based on the star-flash air interface clock. Thus, the timestamps in each media stream are based on a unified time reference, which helps improve synchronization efficiency.

[0004] Firstly, a data transmission method is provided, applied to a transmitting device. It should be noted that, in this application, the term "transmitting device" can refer to the transmitting device itself, or to a chip, functional module, or integrated circuit within the transmitting device that performs the method provided in the embodiments of this application; no specific limitation is made in this application.

[0005] The data transmission method applied to the transmitting device includes the following steps: First, acquiring first media data. The first media data includes multiple media streams and a first timestamp for each media stream. The first timestamp is used to synchronize the multiple media streams. Second, correcting the first timestamps of each media stream based on the StarSpark air interface clock to obtain second media data. The StarSpark air interface clock is the clock used for time synchronization in StarSpark technology. Third, transmitting the second media data.

[0006] Based on this scheme, StarScan technology is a short-range wireless transmission technology characterized by low latency, high-precision synchronization, and anti-interference. Specifically, the clocks of the transmitting and receiving devices (i.e., the StarScan air interface clock) are highly synchronized. Therefore, before transmitting the first media data, the transmitting device corrects the first timestamps of each media stream in the first media data based on the StarScan air interface clock, ensuring high-precision alignment of the media streams on the timeline. This facilitates high-precision synchronization of the media streams in the first media data, improving data synchronization efficiency. Furthermore, the transmitting device only needs to send the first timestamps of each media stream to the receiving device; it does not need to transmit clock correlation information to achieve high-precision synchronization of the media streams. This results in lower data transmission latency and jitter, and higher data transmission efficiency.

[0007] In conjunction with the data transmission method provided in the first aspect, in some possible implementations, the first media data also includes non-time-series media data. The transmission of the second media data includes: encapsulating the second media data based on a first format. The first format supports the encapsulation and transmission of non-time-series media data. The encapsulated second media data is then transmitted. Based on this scheme, using a first format that supports both time-series and non-time-series media to encapsulate the second media data can broaden the application scenarios of the data transmission method and better meet the diverse data transmission needs of businesses.

[0008] In conjunction with the data transmission method provided in the first aspect, in some possible implementations, the first format is the International Organization for Standardization Base Media File Format (ISO BMFF). This approach helps avoid transmission or parsing problems caused by format incompatibility, thus improving data transmission reliability. In some possible implementations, the method also includes adding a second timestamp to the data packets of each media stream based on a StarScan air interface clock when encapsulating the second media data. The second timestamp indicates the order of data packets in the media stream. Based on this approach, by using the second timestamp based on the StarScan air interface clock to indicate the order of data packets in the media stream, accurate identification of the order of data packets in the media stream can be facilitated, efficiently achieving data synchronization during transmission.

[0009] In conjunction with the data transmission method provided in the first aspect, in some possible implementations, the encapsulated second media data includes: encapsulated second media data transmitted based on a non-IP protocol. Based on this scheme, data can be transmitted directly at the physical layer without the need for complex processing at the network and transport layers, which helps reduce transmission latency and improve data transmission efficiency. Furthermore, non-IP protocols can adapt to more network environments and transmission requirements, which is beneficial for improving system compatibility and scalability.

[0010] In conjunction with the data transmission method provided in the first aspect, in some possible implementations, the first media data includes a first media stream, and the interval between adjacent frames in the first media stream is a first interval. The time base for the first timestamp is a first clock, and the clock frequency of the first clock is a first frequency. The clock frequency of the StarSpark air interface clock is a second frequency. Correcting the first timestamp of each media stream based on the StarSpark air interface clock includes: correcting the first timestamp of the first media stream based on the first interval, the first frequency, and the second frequency. Based on this scheme, the first timestamp can be accurately corrected using the StarSpark air interface clock, thereby improving data synchronization efficiency and transmission efficiency.

[0011] Based on the data transmission method provided in the first aspect, in some possible implementations, the first frequency is P1, the second frequency is P2, the first timestamp before correction is T1, and the first timestamp after correction is T2; P1, P2, T1, and T2 satisfy the formula T2 = T1 * P2 / P1. Based on this scheme, the correction of the first timestamp can be reliably and stably achieved through a clear and specific formula.

[0012] Combining the data transmission method provided in the first aspect, in some possible implementations, multiple media streams can include any combination of the following: video streams, audio streams, sensor timing data, and real-time caption streams. Based on this scheme, the transmission needs of various types of media streams can be met, and its application scenarios are quite broad.

[0013] In conjunction with the data transmission method provided in the first aspect, in some possible implementations, the first timestamp includes any one or more of the following: decoding timestamp, displaying timestamp.

[0014] In conjunction with the data transmission method provided in the first aspect, some possible implementations include a second media stream and a first timestamp of the second media stream in the first media data. The second media data also includes the association information between the time base of the first timestamp in the second media stream and the StarScan air interface clock. Based on this scheme, it can be well compatible with existing synchronization mechanisms, which is beneficial for improving the stability and efficiency of synchronization.

[0015] Combining the data transmission method provided in the first aspect, in some possible implementations, the associated information is a Real-Time Transport Control Protocol (RTCP) message. Based on this scheme, RTCP is a standard protocol widely used in real-time data transmission, with mature specifications and broad support. Using it as the carrier of associated information can fully utilize the processing capabilities of existing network devices and systems for RTCP, ensuring reliable transmission and accurate parsing of the associated information, and reducing implementation costs and complexity.

[0016] Secondly, a data transmission method is provided, comprising: receiving second media data. The second media data includes multiple media streams and a first timestamp for each media stream. The first timestamp is used to synchronize the multiple media streams. The time base of the first timestamp is a StarSpark air interface clock; the StarSpark air interface clock is a clock used for time synchronization in StarSpark technology. The multiple media streams are processed based on the StarSpark air interface clock and the first timestamp.

[0017] Based on this scheme, StarScan technology is a short-range wireless transmission technology characterized by low latency, high-precision synchronization, and anti-interference. Specifically, the clocks of the transmitting and receiving devices (i.e., the StarScan air interface clock) are highly synchronized. Therefore, when media data includes multiple media streams and their first timestamps (time base being the StarScan air interface clock), the receiving device can achieve high-precision synchronization of each media stream based on the first timestamp, improving synchronization efficiency. Furthermore, the receiving device only needs to process the first timestamp of each media stream based on the StarScan air interface clock, without needing to process clock association information, resulting in lower data transmission latency and jitter, and higher data transmission efficiency.

[0018] In conjunction with the data transmission method provided in the second aspect, some possible implementations include processing one or more of the following: decoding and display. Based on this scheme, the receiving device can achieve high-precision synchronous decoding and display of each media stream based on the first timestamp and the star-flash air interface clock, which is beneficial to improving the user experience.

[0019] In conjunction with the data transmission method provided in the second aspect, some possible implementations include a second timestamp in the second media data. The second timestamp indicates the order of data packets within the media stream. The method further includes determining the order of data packets in the media stream based on a star-studded air interface clock. Based on this scheme, the receiving device can accurately identify the order of data packets in the media stream based on the star-studded air interface clock and the second timestamp, thereby efficiently achieving data synchronization during transmission.

[0020] In conjunction with the data transmission method provided in the second aspect, some possible implementations include non-time-series media data in the second media data. The second media data is encapsulated using a first format. The first format supports the encapsulation and transmission of non-time-series media data.

[0021] In conjunction with the data transmission method provided in the second aspect, among some possible implementations, the first format is BMFF.

[0022] In conjunction with the data transmission method provided in the second aspect, in some possible implementations, the second media data is transmitted based on a non-IP protocol.

[0023] In conjunction with the data transmission method provided in the second aspect, in some possible implementations, the media stream includes any of the following types: video stream, audio stream, sensor timing data, and real-time caption stream.

[0024] In conjunction with the data transmission method provided in the second aspect, in some possible implementations, the first timestamp includes any one or more of the following: decoding timestamp, displaying timestamp.

[0025] In conjunction with the data transmission method provided in the second aspect, in some possible implementations, the second media data also includes a second media stream, a first timestamp of the second media stream, and association information between the time base of the first timestamp in the second media stream and the StarSpark air interface clock. The method further includes: correcting the first timestamp of the second media stream based on this association information and the StarSpark air interface clock.

[0026] In conjunction with the data transmission method provided in the second aspect, in some possible implementations, the associated information is an RTCP message.

[0027] Thirdly, a data transmission system is provided, comprising: an access layer and a physical layer, an encapsulation layer, and a transport layer. The access layer and physical layer are used to acquire first media data. The first media data includes multiple media streams and a first timestamp for each media stream. The first timestamp is used to synchronize the multiple media streams. The access layer and physical layer are also used to provide a StarSpark air interface clock, which is the clock used for time synchronization in StarSpark technology. The encapsulation layer is used to correct the first timestamp of each media stream based on the StarSpark air interface clock to obtain second media data. The transport layer is used to transmit the second media data.

[0028] Fourthly, a data transmission apparatus is provided, comprising a plurality of interacting modules for implementing the method of either the first or second aspect.

[0029] Fifthly, an electronic device is provided, comprising one or more processors. The one or more processors are configured to execute computer programs or instructions to implement the methods of either the first or second aspect.

[0030] A sixth aspect provides a computer-readable storage medium including a computer program or instructions that, when executed, cause the method of any one of the first or second aspects to be implemented.

[0031] A seventh aspect provides an electronic device including a memory and one or more processors. The memory is used to store computer programs or instructions. The one or more processors are used to execute the computer programs or instructions in the memory, causing the electronic device to perform the methods of either the first or second aspect.

[0032] Eighthly, a computer program product is provided, comprising a computer program or instructions that, when executed, cause the method of any one of the first or second aspects to be implemented.

[0033] A ninth aspect provides a chip device including a processor and a memory. The processor is configured to invoke a computer program or computer instructions stored in the memory to cause the processor to perform the methods described in either the first or second aspect. Optionally, the processor is coupled to the memory via an interface.

[0034] It should be understood that the second to ninth aspects of this application are consistent with or correspond to the technical solutions of the first aspect of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be repeated here. Attached Figure Description

[0035] Figure 1 is a schematic diagram of a streaming media synchronization mechanism.

[0036] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0037] Figure 3 is a flowchart illustrating a data transmission method provided in an embodiment of this application;

[0038] Figure 4 is a schematic diagram of a first media data provided in an embodiment of this application;

[0039] Figure 5 is a schematic diagram of a clock provided in an embodiment of this application;

[0040] Figure 6 is a schematic diagram of a data transmission method provided in an embodiment of this application;

[0041] Figure 7 is a schematic diagram of another data transmission method provided in an embodiment of this application;

[0042] Figure 8 is a schematic diagram of another data transmission method provided in an embodiment of this application;

[0043] Figure 9 is a schematic diagram of the architecture of a data transmission system provided in an embodiment of this application;

[0044] Figure 10 is a schematic diagram of a data transmission device provided in an embodiment of this application. Detailed Implementation

[0045] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0046] References to "one embodiment" or "some embodiments" as described in this application mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0047] Furthermore, in the embodiments of this application, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as an "example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the term "example" is intended to present concepts in a concrete manner. In the embodiments of this application, "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably, and it should be noted that their intended meanings are consistent unless their distinction is emphasized.

[0048] The application scenarios described in this application are for the purpose of more clearly illustrating the technical solutions of this application, and do not constitute a limitation on the technical solutions provided in this application. As those skilled in the art will know, with the emergence of new business scenarios, the technical solutions provided in this application are also applicable to similar technical problems.

[0049] To facilitate understanding, the background proposed in the embodiments of this application will be introduced below.

[0050] During media data transmission, synchronization mechanisms ensure that various types of media streams are correctly synchronized on the receiving device. For example, media data includes video media streams (also known as video streams) and audio media streams (also known as audio streams). Synchronization mechanisms ensure that the video and audio streams are correctly decoded and played synchronously on the receiving device.

[0051] The following are some commonly used synchronization mechanisms as examples.

[0052] The synchronization mechanism of a digital television system is a synchronization mechanism for one-way digital audio and video broadcasting systems, mainly implemented through the following steps. First, the transmitting device encodes the audio and video streams into elementary streams (ES). Then, the transmitting device packages the ES to generate a packetized elementary stream (PES). Next, the transmitting device inserts a decoding time stamp (DTS) / presentation time stamp (PTS) into the header of the PES packets. The time base of the DTS / PTS in the audio stream is based on the audio sampling clock used to describe the audio sampling rate, while the time base of the DTS / PTS in the video stream is based on the video sampling clock used to describe the video frame rate. Then, the transmitting device multiplexes the audio and video streams, packages the PES into transport stream (TS) packets, and inserts a program clock reference (PCR) timestamp into the TS packets. The time base of the PCR timestamp is the system time clock (STC). The STC is a key clock signal in the digital television broadcasting system, generated by the broadcasting equipment. The PCR timestamp is used by the receiving device to recover the transmitter's STC via a phase-locked loop. During multiplexing, the transmitting device unifies the DTS / PTS in the audio and video streams to the STC. Finally, the receiving device recovers the transmitter's STC based on the PCR timestamp and completes decoding and display based on the DTS / PTS in the audio and video streams, thereby ensuring synchronization of audio and video stream decoding and display.

[0053] However, in the synchronization mechanism of digital television systems, the transmitting device inserts a PCR timestamp into the TS packet, and the receiving device introduces additional jitter when processing the PCR timestamp, resulting in reduced data transmission efficiency.

[0054] Streaming media synchronization is a synchronization mechanism for Internet Protocol (IP) networks. In IP networks, media data transmission is based on the Real-Time Transport Protocol (RTP). Specifically, different types of media streams are transmitted separately, each using an independent time base (called a local clock). These time bases are then uniformly associated with the Network Time Protocol (NTP) clock via the Real-Time Transport Control Protocol (RTCP). NTP is the protocol responsible for time synchronization in IP networks, and the NTP clock is a globally unified absolute clock within the IP network. The timestamp carried in RTP messages is based on the local clock, while RTCP messages indicate the association information between the local clock and the NTP clock. In other words, the sending device needs to send additional RTCP messages to indicate the association information between each local clock and the NTP clock, thereby achieving multi-stream synchronization at the receiving device.

[0055] For example, please refer to Figure 1, which is a schematic diagram of a streaming media synchronization mechanism. As shown in Figure 1, in the streaming media synchronization mechanism, the sending device performs advanced video coding (AVC / H264) on the video data and advanced audio coding (AAC) on the audio data. In the TS encapsulation stage after encoding, the sending device inserts DTS / PTS into the TS packets of the video data based on STC1, and inserts DTS / PTS into the TS packets of the audio data based on STC2. In the RTP transmission stage, the sending device indicates the association information between STC1 and the NTP clock, and the association information between STC2 and the NTP clock, through RTCP messages. The NTP clock is determined by the transmission control protocol (TCP) and the IP protocol.

[0056] As can be seen, in the streaming media synchronization mechanism, to achieve synchronization, the sending device needs to send clock correlation information (i.e., RTCP messages), and the receiving device needs to receive and process the clock correlation information. This process introduces additional latency and jitter, leading to a decrease in data transmission efficiency.

[0057] To address the aforementioned issues, embodiments of this application provide a data transmission method, apparatus, and electronic device. Utilizing the high-precision synchronization characteristic of the star-flash air interface clock in star-flash technology between the transmitting and receiving devices, the transmitting device corrects the timestamps in the media stream based on the star-flash air interface clock. Thus, the timestamps in each media stream are based on a unified time reference, which helps improve synchronization efficiency.

[0058] The technical solutions in the embodiments of this application will be clearly and completely described below.

[0059] It should be noted that the technical solutions in the embodiments of this application can be applied to electronic devices (or nodes, network nodes, devices, etc., which are not limited here) in IP networks, such as terminal devices, network devices, etc.

[0060] The terminal device can be a device or apparatus with a chip, or a device or apparatus with integrated circuitry, or a chip, chip system, module, or control unit in the aforementioned devices or apparatus; this application does not impose any specific limitations. It should be noted that, in this application, when referring to a terminal device, it can refer to the terminal device itself, or to the chip, functional module, or integrated circuit in the terminal device that performs the method provided in this application; this application does not impose any specific limitations.

[0061] Examples of terminal devices include: mobile phones, tablets, laptops, PDAs, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in vehicle-to-everything (V2X) communication, wireless terminals in self-driving cars, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, and wireless terminals in smart homes. For example, a wireless terminal in self-driving cars could be a drone, helicopter, or airplane. Similarly, a wireless terminal in V2X communication could be an in-vehicle device, a complete vehicle system, an in-vehicle module, a vehicle, or a ship. A wireless terminal in industrial control could be a camera, a robot, or a robotic arm. A wireless terminal in a smart home could be a television, air conditioner, robot vacuum cleaner, speaker, or set-top box.

[0062] Network equipment can refer to equipment used to build and manage wireless networks. Network equipment can be a device or apparatus with a chip, or a device or apparatus with integrated circuitry, or a chip, chip system, module, or control unit within the aforementioned devices or apparatus; this application does not impose any specific limitations. It should be noted that, in this application, the term "network equipment" can refer to either the network equipment itself or the chip, functional module, or integrated circuit within the network equipment that performs the methods provided in this application; this application does not impose any specific limitations.

[0063] Examples of network devices include: evolved Node B (eNB), radio network controller (RNC), Node B (NB), base station controller (BSC), base transceiver station (BTS), home base station (e.g., home evolved Node B, or home Node B, HNB), baseband unit (BBU), access point (AP) in a Wi-Fi system, macro base station, micro base station, wireless relay node, donor node, wireless controller in a CRAN scenario, wireless backhaul node, transmission point (TP), or transmission and receiving point (TRP), etc.

[0064] For example, please refer to FIG2, which is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. As shown in FIG2, the electronic device 200 may include one or more processors 201 (FIG. 2 uses one processor as an example). Optionally, the electronic device 200 may also include one or more memories 202 coupled to the processor 201 (FIG. 2 uses one memory as an example, indicated by a dashed box), the memory 202 is used to store computer programs or instructions and / or data, and the processor 201 is used to execute the computer programs or instructions and / or data stored in the memory 202, so that the methods or steps in the embodiments of this application are executed.

[0065] Alternatively, the memory 202 can be integrated with the processor 201, or it can be set separately.

[0066] Optionally, the electronic device 200 may also include a transceiver 201 for receiving and / or sending messages. For example, a processor 201 may be used to control the transceiver 201 to receive and / or send messages.

[0067] In the embodiments of this application, the processor may include one or more of the following: central processing unit (CPU), digital signal processor (DSP), microprocessor unit (MPU), microcontroller unit (MCU), graphics processing unit (GPU), field programmable gate array (FPGA), artificial intelligence processor (AI processor), or neural processing unit (NPU).

[0068] Memory may include one or more of the following: random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), phase-change memory (PCM), resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), cache, register, read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM), and hard disk.

[0069] Based on the above background and application description of this solution, the data transmission method, apparatus, and electronic equipment provided in the embodiments of this application will be described below.

[0070] The data transmission method, apparatus, and electronic device provided in this application are based on star flash technology. For ease of understanding, star flash technology will be introduced below.

[0071] Starflash technology is a short-range wireless communication technology. Based on advanced wireless communication protocols and modulation and demodulation techniques, it employs efficient spectrum utilization to ensure stable and high-speed data transmission, thereby ensuring efficient and stable connections between devices.

[0072] The low latency of StarScan technology is one of its core advantages. StarScan employs efficient synchronization algorithms and protocols to ensure that the clocks of the transmitting and receiving devices remain highly consistent. Specifically, StarScan technology achieves a synchronization accuracy of less than 1 microsecond through ultra-short frames, multi-point synchronization, two-way authentication encryption, and cross-layer scheduling optimization, thereby ensuring the accuracy and reliability of data transmission.

[0073] In other words, in the Starflash technology, the transmitting and receiving devices are highly synchronized, meaning the clocks of the transmitting and receiving devices are also highly synchronized. Based on this characteristic, this application provides a data transmission method, which will be described in detail below.

[0074] For ease of explanation, the synchronization clock between the transmitting and receiving devices in the spark technology will be referred to as the spark time clock (SPC) in the following embodiments, and will not be elaborated further.

[0075] Please refer to Figure 3, which is a flowchart illustrating a data transmission method provided in an embodiment of this application. As shown in Figure 3, the method may include the following steps.

[0076] S301, The transmitting device acquires the first media data.

[0077] In the embodiments of this application, when referring to the transmitting device, it may refer to the transmitting device itself, or to the chip, functional module or integrated circuit in the transmitting device that performs the method provided in the embodiments of this application. The specific meaning is not limited in this application.

[0078] The first type of media data can be time-series media data or non-time-series media data. Time-series media data refers to media data that needs to be processed in chronological order during decoding, playback / display, such as video, audio, and sensor time-series data. Non-time-series media data refers to media data that does not depend on chronological order during decoding, playback / display, such as text and images.

[0079] The first media data may include multiple media streams with a first timestamp. The first timestamp is used to synchronize the multiple media streams in the first media data.

[0080] For example, referring to Figure 4, the first media data includes an audio stream 401 and a video stream 402. The audio stream 401 includes multiple audio frames (Figure 4 uses two audio frames as an example, namely audio frame 411 and audio frame 421), and the video stream 402 includes multiple video frames (Figure 4 uses two video frames as an example, namely video frame 412 and video frame 422). Each audio frame in the audio stream and each video frame in the video stream carries a first timestamp for synchronization. The first timestamp in the audio stream is based on a time base T, and the first timestamp in the video stream is based on a time base S.

[0081] As shown in Figure 4, audio frame 411 has a first timestamp t1, audio frame 421 has a first timestamp t2, video frame 412 has a first timestamp s1, and video frame 422 has a first timestamp s2. As an example, the first timestamps t1 and s1 are aligned on an absolute clock, and the first timestamps t2 and s2 are also aligned on an absolute clock. Therefore, synchronization of audio frame 411 and video frame 412 can be achieved using t1 and s1, and synchronization of audio frame 421 and video frame 422 can be achieved using t2 and s2.

[0082] In this embodiment, the first timestamp can be DTS, PTS, or both; no limitation is made here. The transmitting device can add timestamps for multiple media streams to the first media data based on the methods described in the foregoing embodiments or other methods, such as adding DTS / PTS to video data and audio data respectively based on a streaming media synchronization mechanism. The time standards on which the first timestamps of different media streams are based can be the same or different (as shown in Figure 4, the time base T and time base S can be the same or different); this embodiment does not limit this.

[0083] In the embodiments of this application, the sending device acquiring the first media data may refer to receiving the first media data sent by an external device, or it may refer to the sending device generating the first media data; no limitation is made here.

[0084] S302. The transmitting device corrects the first timestamp of each media stream based on the star-flash air interface clock to obtain the second media data.

[0085] As described in the foregoing embodiments, the StarSpark air interface clock is the synchronization clock between the transmitting and receiving devices in StarSpark technology, or in other words, the clock used for time synchronization in StarSpark technology.

[0086] The transmitting device can correct the time base of the first timestamp in each media stream to the StarSpark air interface clock, thereby completing the correction of each first timestamp.

[0087] For example, suppose the first media data includes a first media stream, the time base of the first timestamp T1 in the first media stream is a first clock, and the clock frequency of the first clock is a first frequency P1. The clock frequency of the StarScan air interface clock is a second frequency P2. The corrected first timestamp is T2. Then the transmitting device can correct the first timestamp of the first media stream based on the first interval, the first frequency, and the second frequency, such as calculating T1*P2 / P1 to obtain T2.

[0088] The following are specific examples.

[0089] The following example, using a low-latency coded scenario without bidirectional predictive frames (B-frames), illustrates the process of correcting the first timestamp. A B-frame is a frame type in video coding that does not contain complete image information but stores differences between it and preceding and following frames. Therefore, when a media stream contains B-frames, the decoding order and display order are different, and the DTS and PTS are not equal. When a media stream does not contain B-frames, the decoding order and display order are usually the same, and the DTS and PTS are equal.

[0090] In other words, in low-latency coding scenarios without B-frames, DTS and PTS are equal. For ease of explanation, the following uses the first timestamp as the PTS as an example to illustrate the process of correcting the first timestamp.

[0091] Please refer to Figure 5, which is a schematic diagram of a clock provided in an embodiment of this application. As shown in Figure 5, the system reference clock STC has a clock frequency of 90 kHz, the audio sampling clock has a frequency of 48 kHz, the video sampling clock has a frequency of 90 kHz, and the StarScan air interface clock has a frequency of 48 kHz. The frame rate of the audio data (also referred to as the audio stream) is 50 fps, and the frame rate of the video data (also referred to as the video stream) is 30 fps. The audio sampling clock and video sampling clock are obtained by default based on the system reference clock STC.

[0092] It should be understood that in audio data, the interval between adjacent audio frames is 1 / 50 = 0.02, the PTS unit for audio frames is 1 / 48000, and the PTS interval for audio frames is 0.02 / (1 / 48000) = 960. Similarly, the interval between adjacent video frames is 1 / 30, the PTS unit for video frames is 1 / 90000, and the PTS interval for video frames is (1 / 30) / (1 / 90000) = 3000. Therefore, we can obtain the PTS value sequence of audio frames based on the system reference clock (STC), and the PTS value sequence of video frames based on the system reference clock (STC), as shown in Figure 5.

[0093] The frequency of the StarSpark over-the-air clock is 48kHz. Therefore, by correcting the PTS value sequences of the audio and video frames based on the StarSpark over-the-air clock, we can obtain the PTS value sequences of the audio and video frames based on the StarSpark over-the-air clock, as shown in Figure 5. This completes the correction process for the first timestamp based on the StarSpark over-the-air clock.

[0094] It should be noted that since the system reference clock STC and the StarScan air interface clock are from different sources, drift and deviation are inevitable. Therefore, the PTS value calculated above is the nominal value. In practical applications, the PTS value (shown in bold in Figure 5) will gradually deviate from the nominal value and will not increase strictly according to the intervals calculated above.

[0095] To eliminate the time accumulation effect caused by drift and deviation from different clock sources, for example, a counter Count0 can be configured for the system reference clock STC and a counter Count1 can be configured for the star flash over-the-air clock.

[0096] Assuming that the two counters are initialized to 0 at the same time, and the counter values ​​of the two clocks at time t are Count0(t) and Count1(t) respectively, the cumulative effect caused by the drift and deviation of different clock sources can be eliminated by the following formula (1) (e.g., eliminating the influence of PTS0(t) to obtain PTS1(t)).

[0097] PTS1(t)=PTS0(t)*Count1(t) / Count0(t) Formula (1)

[0098] S303, The transmitting device transmits the second media data to the receiving device.

[0099] In some possible implementations, the sending device may first encapsulate the second media data and then transmit the encapsulated second media data to the receiving device.

[0100] In this embodiment, the transmitting device can encapsulate the second media data using an encapsulation format that supports both time-series and non-time-series media data (referred to as the first format). The encapsulation format supporting both time-series and non-time-series media data can be an encapsulation format conforming to the Moving Picture Experts Group (MMT) standard, such as ISO BMFF, or BMFF-based Moving Picture Experts Group Part 4, Sub-14 (MP4), etc. This broadens the application scenarios.

[0101] In some possible implementations, the transmitting device can also transmit the second media data to the receiving device based on a simple and efficient non-internet protocol (nonIP) designed for short-range scenarios. NonIP is a communication protocol that does not rely on the IP protocol stack, does not use traditional IP addresses for data transmission, and features low power consumption, high security, high transmission efficiency, and a simple network architecture. Therefore, transmitting second media data based on nonIP can reduce data transmission overhead and improve data transmission efficiency.

[0102] In some possible implementations, the transmitting device can also add a second timestamp to the data packets of each media stream based on the StarScan air interface clock when encapsulating the second media data. The second timestamp is used to indicate the order of the data packets in the media stream. In this way, the receiving device can directly determine the order of each data packet in the media stream based on the StarScan air interface clock, thereby effectively avoiding out-of-order anomalies during data transmission and improving data transmission efficiency.

[0103] In some possible implementations, the data transmission method provided in this application embodiment can also support mixed transmission of media streams with different encapsulation formats. Please refer to Figure 6, which is a schematic diagram of a data transmission method provided in an embodiment of this application. As shown in Figure 6, the transmitting device includes an encoding layer 601, an encapsulation layer 602, a transmission layer 603, an access layer, and a physical layer 604. Among them, the access layer and the physical layer 604 are used to provide a StarScan air interface clock.

[0104] For video data, the encoding layer 601 can be used to encode the video data using H.264. The encapsulation layer 602 can be used to correct the DTS / PTS in the video data based on the StarFlash air interface clock, and to encapsulate the corrected video data based on ISO BMFF. The transport layer 603 can be used to send the video data to the receiving device via nonIP.

[0105] For audio data, encoding layer 601 can be used to perform AAC encoding on the audio data. Encoding layer 602 can be used to generate DTS / PTS for each data packet in the audio data based on the local system reference clock, and encapsulate the audio data into TS packets. Transport layer 603 can be used to associate the local system reference clock with the StarScan air interface clock via RTCP messages, and send the audio data to the receiving device via TCP / IP.

[0106] In other words, the data transmission method provided in this application supports the transmission of second media data after encapsulation in the first format, and also supports the mixed transmission of media streams with different encapsulation formats (including the first format). This application does not specifically limit this.

[0107] S304. The receiving device processes multiple media streams based on the Star Flash air interface clock and the first timestamp.

[0108] It should be understood that the first timestamp in S304 refers to the first timestamp after the transmitting device has corrected its clock based on the Star Flash air interface clock.

[0109] The processing may include decoding, display, etc. For example, the receiving device can decode and display the media stream based on the StarSpark air interface clock and the first timestamp in the media stream. It should be understood that the first timestamp is corrected by the sending device based on the StarSpark air interface clock, so the receiving device can ensure synchronization when decoding and displaying based on the StarSpark air interface clock.

[0110] In some possible implementations, the receiving device can also determine the order of data packets in the media stream based on the second timestamp in the second media data and the star-flash air interface clock. This effectively avoids out-of-order transmission and other anomalies, thus improving data transmission efficiency. For details, please refer to the aforementioned introduction to the second timestamp; it will not be repeated here.

[0111] The data transmission method provided in this application embodiment will be described again from the perspective of the system architecture of the sending device, taking Figure 7 as an example.

[0112] Please refer to Figure 7, which is a schematic diagram of a data transmission method provided in an embodiment of this application. As shown in Figure 7, the transmitting device includes an encoding layer 701, an encapsulation layer 702, a transport layer 703, an access layer, and a physical layer 704. The access layer and physical layer 704 are used to provide a StarScan air interface clock. The encoding layer 701 can be used to encode video data using H.264 and audio data using AAC. The encapsulation layer 702 can be used to correct DTS / PTS in video and audio data based on the StarScan air interface clock. The transport layer 703 can be used to add a transport layer timestamp (i.e., a second timestamp) to the video and audio data based on the StarScan air interface clock, encapsulate the video and audio data based on BMFF, and send the encapsulated data to the receiving device via nonIP.

[0113] The data transmission method provided in this application embodiment will be described again from the perspective of the interaction between the sending end device and the receiving end device.

[0114] Please refer to Figure 8, which is a schematic diagram of another data transmission method provided in this application embodiment. As shown in Figure 8, the transmitting device 801 uses a star-flash air interface clock as a time base, and the receiving device 802 also uses a star-flash air interface clock as a time base. The star-flash air interface clock used by the transmitting device and the star-flash air interface clock used by the receiving device are synchronized based on star-flash technology.

[0115] After receiving the media stream (also known as media data), the transmitting device 801 can encode the media stream, such as performing H.264 encoding on video data (also known as video stream) and AAC encoding on audio data (also known as audio stream). Then, the transmitting device 801 can correct the DTS / PTS of the media stream based on the StarSpark over-the-air clock. After correction, the transmitting device 801 can add a transport layer timestamp to the data packets of the media stream based on the StarSpark over-the-air clock, encapsulate them using BMFF, and send them to the receiving device 802 via a lightweight nonIP transport protocol.

[0116] It should be understood that the StarSpark air interface clock used by the transmitting device 801 is synchronized with the StarSpark air interface clock used by the receiving device 802. Therefore, the receiving device 802 can directly determine the order of data packets in the media stream based on the StarSpark air interface clock, thereby effectively avoiding out-of-order issues during media stream transmission. Furthermore, the receiving device 802 can also directly decode and play the media stream using the StarSpark air interface clock, achieving media stream synchronization with lower overhead. It can be seen that, based on the data transmission method provided in this application embodiment, the transmitting device can achieve synchronization of timing media data on the receiving device without transmitting clock association information. This reduces latency and jitter and helps reduce overhead. Moreover, the encapsulation is based on the ISO BMFF standard file format, supporting the transmission of both timing and non-timing media, and has a wide range of application scenarios. Additionally, the use of a lightweight nonIP transport layer results in lower overhead.

[0117] This application also provides a data transmission system. Please refer to Figure 9, which is a schematic diagram of the architecture of a data transmission system provided in this application embodiment. As shown in Figure 9, the data transmission system 900 includes: an access layer and a physical layer 901, an encapsulation layer 902, and a transport layer 903.

[0118] The access layer and physical layer 901 are used to acquire the first media data. The access layer and physical layer 901 are also used to provide a satellite radio interface clock. The encapsulation layer 902 is used to correct the first timestamp of each media stream based on the satellite radio interface clock to obtain the second media data. The transport layer 903 is used to transmit the second media data.

[0119] In some possible implementations, the data transmission system 900 may also include an encoding layer and a decoding layer 904. The encoding layer and decoding layer 904 are used to encode and decode media data.

[0120] For an introduction to the first media data, the StarScan air interface clock, the first timestamp, and the second media data, please refer to the descriptions in the foregoing embodiments; they will not be repeated here.

[0121] This application also provides a data transmission device, which may include multiple interacting modules for implementing any of the methods described in the above method embodiments.

[0122] For example, please refer to FIG10, which is a schematic diagram of a data transmission device provided in an embodiment of this application. As shown in FIG10, the data transmission device 1000 includes a processing module 1001 and a transceiver module 1002.

[0123] In some possible implementations, processing module 1001 acquires first media data. The first media data includes multiple media streams and a first timestamp for each media stream. The first timestamp is used to synchronize the multiple media streams. Processing module 1001 corrects the first timestamp of each media stream based on the StarSpark air interface clock to obtain second media data. The StarSpark air interface clock is the clock used for time synchronization in StarSpark technology. Transceiver module 1002 is used to transmit the second media data.

[0124] In some possible implementations, the transceiver module 1002 is used to receive second media data. The second media data includes multiple media streams and a first timestamp for each media stream. The first timestamp is used to synchronize the multiple media streams. The processing module 1001 is used to process the multiple media streams based on the StarSpark air interface clock and the first timestamp. The StarSpark air interface clock is the clock used for time synchronization in StarSpark technology.

[0125] It should be noted that some optional features in the various embodiments of this application may not depend on other features in certain scenarios, or may be combined with other features in certain scenarios, without limitation.

[0126] The solutions in the various embodiments of this application can be used in reasonable combinations, and the explanations or descriptions of various terms, similar operations, or steps appearing in the embodiments can be referenced or explained to each other in the various embodiments, without limitation.

[0127] This application also provides a system, including one or more of the above-described devices, apparatuses, computer-readable storage media, computer program products, chips, or chip systems.

[0128] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the explanation of the relevant content and beneficial effects of any of the devices, equipment, and media provided above can be referred to the corresponding method embodiments provided above, and will not be repeated here.

[0129] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0130] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0131] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0132] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the essential contribution of the technical solution of this application, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0133] The above embodiments are merely illustrative of the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to depart from the scope of the technical solutions of the embodiments of this application.

Claims

1. A data transmission method, characterized in that, include: Acquire first-hand media data; The first media data includes multiple media streams and a first timestamp for each of the media streams; The first timestamp is used to synchronize the multiple media streams; The second media data is obtained by correcting the first timestamp of each media stream based on the StarSpark air interface clock; the StarSpark air interface clock is the clock used for time synchronization in StarSpark technology. Transmit the second media data.

2. The data transmission method according to claim 1, characterized in that, The first media data also includes non-time-series media data; The transmission of the second media data includes: The second media data is encapsulated based on a first format; the first format supports the encapsulation and transmission of the non-time-series media data. Transmit the encapsulated second media data.

3. The data transmission method according to claim 2, characterized in that, The first format is the basic media file format defined by the International Organization for Standardization.

4. The data transmission method according to claim 2 or 3, characterized in that, The method further includes: When encapsulating the second media data, a second timestamp is added to the data packets of each media stream based on the StarScan air interface clock; the second timestamp is used to indicate the order of the data packets in the media stream.

5. The data transmission method according to any one of claims 2-4, characterized in that, The transmission of the encapsulated second media data includes: The encapsulated second media data is transmitted based on a non-IP protocol.

6. The data transmission method according to any one of claims 1-5, characterized in that, The plurality of media streams includes a first media stream, wherein the time base of the first timestamp in the first media stream is a first clock, and the clock frequency of the first clock is a first frequency; The clock frequency of the StarScan air interface clock is the second frequency; The correction of the first timestamp of each media stream based on the star-flash air interface clock includes: The first timestamp in the first media stream is corrected based on the first frequency and the second frequency.

7. The data transmission method according to claim 6, characterized in that, The first frequency is P1, the second frequency is P2, the first timestamp before correction is T1, and the first timestamp after correction is T2; P1, P2, T1, and T2 satisfy the formula T2 = T1 * P2 / P1.

8. The data transmission method according to any one of claims 1-7, characterized in that, The multiple media streams include any of the following: video stream, audio stream, sensor timing data, and real-time caption stream.

9. The data transmission method according to any one of claims 1-8, characterized in that, The first timestamp includes any one or more of the following: a decoded timestamp, a displayed timestamp.

10. The data transmission method according to any one of claims 1-9, characterized in that, The first media data also includes a second media stream and a first timestamp of the second media stream; the second media data also includes the association information between the time base of the first timestamp in the second media stream and the StarScan air interface clock.

11. The data transmission method according to claim 10, characterized in that, The associated information is a real-time transmission control protocol message.

12. A data transmission method, characterized in that, include: Receive second media data; The second media data includes multiple media streams and a first timestamp of each media stream; The first timestamp is used to synchronize the multiple media streams, and the time base of the first timestamp is the StarSpark air interface clock; the StarSpark air interface clock is the clock used for time synchronization in StarSpark technology; The multiple media streams are processed based on the StarScan air interface clock and the first timestamp.

13. The data transmission method according to claim 12, characterized in that, The processing includes any one or more of the following: decoding, display.

14. The data transmission method according to claim 12 or 13, characterized in that, The second media data also includes a second timestamp, which is used to indicate the order of data packets in the media stream; The method further includes: The order of data packets in the media stream is determined based on the StarScan air interface clock.

15. The data transmission method according to any one of claims 12-14, characterized in that, The second media data also includes non-time-series media data; the second media data is encapsulated in a first format, which supports the encapsulation and transmission of the non-time-series media data.

16. The data transmission method according to claim 15, characterized in that, The first format is the basic media file format defined by the International Organization for Standardization.

17. The data transmission method according to any one of claims 12-16, characterized in that, The second media data is transmitted based on a non-IP protocol.

18. The data transmission method according to any one of claims 12-17, characterized in that, The media stream includes any of the following types: video stream, audio stream, sensor timing data, and real-time caption stream.

19. The data transmission method according to any one of claims 12-18, characterized in that, The first timestamp includes any one or more of the following: a decoded timestamp, a displayed timestamp.

20. The data transmission method according to any one of claims 12-19, characterized in that, The second media data also includes a second media stream, a first timestamp of the second media stream, and the association information between the time base of the first timestamp in the second media stream and the StarSpark air interface clock; The method further includes: correcting the first timestamp of the second media stream based on the association information and the StarScan air interface clock.

21. The data transmission method according to claim 20, characterized in that, The associated information is a real-time transmission control protocol message.

22. A data transmission system, characterized in that, include: Access layer and physical layer, encapsulation layer and transport layer; The access layer and physical layer are used to acquire the first media data; The first media data includes multiple media streams and a first timestamp for each of the media streams; The first timestamp is used to synchronize the multiple media streams; the access layer and physical layer are also used to provide a StarSpark air interface clock, which is a clock used for time synchronization in StarSpark technology; The encapsulation layer is used to correct the first timestamp of each media stream based on the Star Flash air interface clock to obtain the second media data; The transport layer is used to transmit the second media data.

23. A data transmission device, characterized in that, It includes multiple interacting modules for implementing the method as described in any one of claims 1-11 or the method as described in any one of claims 12-21.

24. An electronic device, characterized in that, It includes one or more processors; the one or more processors are configured to execute computer programs or instructions to implement the method as described in any one of claims 1-11 or the method as described in any one of claims 12-21.

25. A computer-readable storage medium, characterized in that, Includes a computer program or instructions that, when executed, cause the method of any one of claims 1-11 or the method of any one of claims 12-21 to be implemented.

Images