Device and method for supporting audio and video synchronization

Abstract

The present disclosure relates to a device and a method for supporting audio and video synchronization. Specifically, the present disclosure relates to a device and a method for a source device to transmit an audio packet and a video packet which include common time information (a time stamp) to a sink device through an interface such as High-Definition Multimedia Interface (HDMI), DisplayPort, and the like, and the sink device to perform synchronization between audio and video by using the time information.

Description

Device and method for supporting audio and video synchronization

[0001] The present disclosure relates to an apparatus and method for supporting audio and video synchronization. Specifically, the present disclosure relates to an apparatus and method for a sink device to perform synchronization between audio and video by transmitting audio packets and video packets containing a common time stamp to a sink device via an interface such as HDMI (High-Definition Multimedia Interface) or DisplayPort.

[0002]

[0003] The present disclosure relates to an apparatus and method for supporting audio and video synchronization. Recently, technology for transmitting high-definition audio and video data between a source device and a sink device using high-speed digital interfaces such as HDMI (High-Definition Multimedia Interface) and DisplayPort has been widely used.

[0004] However, according to conventional technology, a problem arose where the synchronization between the two streams was not perfectly aligned during the process of handling audio and video transmitted from a source device to a sink device. This tends to occur because video data has a larger volume than audio, requiring more complex and longer pipeline stages to process.

[0005] In addition, the processing method of specific video formats also caused problems. For example, when 24Hz video used in movies was transmitted after being 'doubled' by the source device, there was a problem where video transition noise occurred during this process. This noise was perceived by users as video stuttering.

[0006] Due to these asynchronous issues and video stuttering, there was a problem where the user experience was degraded, especially when users watched movie content through the DisplayPort interface.

[0007]

[0008] To solve the aforementioned problems, the present disclosure provides an apparatus and method for supporting audio and video synchronization.

[0009] The present disclosure provides an apparatus and method for a sink device to perform synchronization between audio and video by using the time information, wherein a source device transmits audio packets and video packets to a sink device via an interface such as HDMI (High-Definition Multimedia Interface) or DisplayPort, by including common time information (time stamp) in the packets.

[0010] The technical problems to be solved in this disclosure are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this disclosure belongs from the description below.

[0011]

[0012] According to various embodiments of the present disclosure, a method of operating a source device is provided, comprising the steps of: generating common time information for video data and audio data; inserting the time information into the video data; inserting the time information into the audio data; and transmitting the video data containing the time information and the audio data containing the time information to a sink device.

[0013] According to various embodiments of the present disclosure, a method for operating a sink device is provided, comprising the steps of: receiving video data containing first time information and audio data containing second time information from a source device; obtaining the first time information from the video data; obtaining the second time information from the audio data; and comparing the first time information and the second time information to synchronize the video data and the audio data.

[0014] According to various embodiments of the present disclosure, a source device comprises: a processor; a memory; and a transceiver, wherein the memory stores instructions for performing operations based on execution by the processor, the operations comprising: generating common time information for video data and audio data; inserting the time information into the video data; inserting the time information into the audio data; and transmitting the video data containing the time information and the audio data containing the time information to a sink device.

[0015]

[0016] To solve the aforementioned problems, the present disclosure may provide an apparatus and method for supporting audio and video synchronization.

[0017] The present disclosure provides a device and method for a sink device to perform synchronization between audio and video by including common time information (time stamp) in audio packets and video packets and transmitting them to a sink device through an interface such as HDMI (High-Definition Multimedia Interface) or DisplayPort.

[0018]

[0019] The drawings attached below are intended to aid in understanding the present disclosure and may provide embodiments of the present disclosure together with the detailed description. However, the technical features of the present disclosure are not limited to specific drawings, and the features disclosed in each drawing may be combined with one another to form new embodiments. Reference numerals in each drawing may denote structural elements.

[0020] FIG. 1 is a block diagram showing a system according to various embodiments of the present disclosure.

[0021] FIG. 2 is a block diagram showing an example of the structure of a Source device and a Sink device according to various embodiments of the present disclosure.

[0022] FIG. 3 is a block diagram showing an example of the structure of a Source device and a Sink device according to various embodiments of the present disclosure.

[0023] FIG. 4 is a diagram showing the initial connection sequence between a source device and a sink device according to the prior art.

[0024] Figure 5 is a diagram illustrating the asynchronous problem that occurs during audio and video transmission according to the prior art and the prior art processing method for it.

[0025] FIG. 6 is a diagram showing a sequence for performing audio and video synchronization using time information (Time Stamp) according to one embodiment of the present disclosure.

[0026] FIG. 7 is a diagram schematically showing the structure of a video frame to which an embodiment of the present disclosure can be applied.

[0027] FIG. 8 is a block diagram illustrating the processing process of video source data in one embodiment of the present disclosure.

[0028] FIG. 9 is a diagram illustrating the Active Video period and the Data Island Period within a video frame according to one embodiment of the present disclosure.

[0029] FIG. 10 is a drawing illustrating an example of a method of operation of a Source device according to various embodiments of the present disclosure.

[0030] FIG. 11 is a drawing illustrating an example of a method of operation of a Sink device according to various embodiments of the present disclosure.

[0031]

[0032] In various embodiments of the present disclosure, "A or B" may mean "only A," "only B," or "both A and B." Alternatively, in various embodiments of the present disclosure, "A or B" may be interpreted as "A and / or B." For example, in various embodiments of the present disclosure, "A, B or C" may mean "only A," "only B," "only C," or "any combination of A, B and C."

[0033] In various embodiments of the present disclosure, a slash ( / ) or a comma used may mean "and / or." For example, "A / B" may mean "A and / or B." Accordingly, "A / B" may mean "only A," "only B," or "both A and B." For example, "A, B, C" may mean "A, B or C."

[0034] In various embodiments of the present disclosure, "at least one of A and B" may mean "only A," "only B," or "both A and B." Additionally, in various embodiments of the present disclosure, the expressions "at least one of A or B" or "at least one of A and / or B" may be interpreted as synonymous with "at least one of A and B."

[0035] Additionally, in various embodiments of the present disclosure, “at least one of A, B and C” may mean “only A,” “only B,” “only C,” or “any combination of A, B and C.” Also, “at least one of A, B or C” or “at least one of A, B and / or C” may mean “at least one of A, B and C.”

[0036]

[0037] FIG. 1 is a block diagram showing a system according to various embodiments of the present disclosure.

[0038] Hereinafter, devices that transmit and receive video, audio, and control data will be collectively referred to as AV (audio / video) systems. Examples of AV systems include HDMI and DisplayPort.

[0039] Referring to FIG. 1, the AV system may include a source device (100) and a sink device (200). In particular, in the AV system, the device that transmits video / audio data corresponds to the source device (100), and the device that receives video / audio data corresponds to the sink device (200). At this time, cables and connectors may be provided as physical devices that connect the two devices to support data transmission and reception.

[0040] Cables and connectors can perform pairing of four channels providing TMDS (Transition Minimized Differential Signaling) data channels and TMDS clock channels. TMDS data channels can be used to transmit video data, audio data, and auxiliary data.

[0041] Additionally, the AV system provides the VESA (Video Electronics Standards Association) Display Data Channel (DDC). The DDC is used for exchanging configuration and status information between source and sink devices. The CEC protocol can provide high-level control functions between various audio-visual products in the user environment and may be used optionally. Furthermore, the optional HDMI Ethernet and Audio Return Channel (HEAC) may provide Ethernet-compatible data networking between the Audio Return Channel (ARC) and connected devices from the opposite direction from the TMDS.

[0042] Video data, audio data, and auxiliary data can be transmitted / received through three TMDS data channels. The TMDS clock typically runs the video pixel rate and is transmitted through the TMDS clock channel. The TMDS clock can be used as a frequency reference for data recovery in the three TMDS data channels at the receiver. At the source device, 8 bits of data per TMDS data channel can be converted into a 10-bit DC-balanced, transition-minimized sequence and transmitted serially at a rate of 10 bits per TMDS clock period.

[0043] To transmit audio data and auxiliary data through TMDS channels, AV systems use a packet structure. To achieve high reliability for audio data and control data, data can be transmitted as 10-bit words generated using BCH error correction codes and error reduction coding.

[0044] The source device can read the E-EDID (Enhanced Extended Display Identification Data) of the DDC (Display Data Channel) sink device to determine the configuration information and available functions of the sink device. The E-EDID may also be referred to as EDID information below.

[0045] The utility line can be used for optional extension functions such as HEAC.

[0046] The source device (100) can receive EDID (Extended Display Identification Data) information from the sink device (200) through a DDC channel. The source device (100) can parse the received EDID information to recognize configuration information and support functions of the sink device (200). The EDID information may include at least one block containing various information regarding the sink device (200).

[0047] In particular, EDID information according to one embodiment of the present invention may include information regarding the function and power supply capability of the sink device (200) in power transmission and reception. The source device (100) recognizes the power transmission / reception capability of the sink device (200) through this EDID information and, accordingly, can transmit power to the sink device (200) or receive power from the sink device (200).

[0048] The source device (100) includes at least one of a display unit (110), a user input interface unit (120), a control unit (180), a transmitter (Tx), a memory unit (140), a storage unit (150), a multimedia unit (160), a power control unit (130), and a power supply unit (170).

[0049] The sink device (200) includes at least one of an EDID EEPROM (210), a power control unit (220), a display unit (230), a user input interface unit (240), a receiver (Rx), a control unit (280), a power supply unit (250), a memory unit (260), and a multimedia unit (270). In the following description, the description of units performing the same operation will not be duplicated.

[0050] The source device (100) represents a physical device that transmits or streams content stored in the storage unit (150) to the sink device (200). The source device (100) can send a request message to the sink device (200) or receive and process a request message received from the sink device (200). The source device (100) can provide a UI that processes a response message sent by the sink device (200) in response to the transmitted request message and delivers it to the user, and if the source device (100) includes a display unit (110), this UI can be provided as a display. Additionally, the source device (100) can request power to be supplied from the sink device (200).

[0051] The sink device (200) receives content from the source device (100) and can send a request message to the source device (100) or process a message received from the source device (100) and send a response message. The sink device (200) can also provide a User Interface (UI) that processes a response message received from the source device (100) and delivers it to a user, and if the sink device (200) includes a display unit, it can provide this UI as a display. Additionally, the sink device (200) can supply power requested by the source device (100) to the source device (100).

[0052] The user input interface unit (120, 240) can receive user action or input, and as an example, the user input interface (120, 240) may correspond to a remote controller, a voice receiving / recognition device, a touch input sensing / receiving device, etc.

[0053] The control unit (180, 280) can control the overall operation of each device. In particular, the control unit (180, 280) can perform communication between the units included in each device and control the operation of each unit.

[0054] The memory unit (140, 260) represents a volatile physical device in which various types of data are temporarily stored.

[0055] A storage unit (150) represents a non-volatile physical device capable of storing various types of data.

[0056] The EDID EEPROM (210) represents an EEPROM that stores EDID information.

[0057] The memory unit (140, 260), storage unit (150), and EDID EEPROM (210) described above all serve the function of storing data, and they may all be collectively referred to as memory units.

[0058] The display unit (110, 230) can display received data or content, data stored in the memory unit, UI, etc., under the control of the control unit (180, 280).

[0059] The multimedia unit (160, 270) can play various types of multimedia. The multimedia unit (160, 270) may be implemented separately from the control unit (180, 280) or may be implemented as a single physical configuration with the control unit (180, 280).

[0060] The power supply unit (170, 250) can supply power required for the operation of the source device (100), the sink device (200), and the units included therein.

[0061] The transmitter (Tx) is a unit equipped in the source device (100) for transmitting and receiving data, and performs data transmission and reception including not only audio / video data but also messages such as commands, requests, actions, and responses between devices.

[0062] The receiver (Rx) is a unit equipped in the sink device (200) for transmitting and receiving data, and performs data transmission and reception including not only audio / video data but also messages such as commands, requests, actions, and responses between devices.

[0063] The power control unit (130, 220) can manage and control power transmission and reception between devices through the transceiver.

[0064] Among the units described above, units other than the transmitter (Rx), receiver (Tx), and control unit (180, 280) may be optionally included in the source device (100) or sink device (200) according to the embodiment and may not correspond to essential component units.

[0065] Previously, power transfer between source and sink devices was not supported in AV systems. As a result, when operating portable devices for extended periods, there was the inconvenience of having to constantly connect an external power cable to ensure optimal operation. To resolve this inconvenience, this specification proposes a method to ensure optimal operation of the AV system without the need for a separate external device by enabling a wired interface in the AV system to support a power transfer function.

[0066] For the sake of convenience of explanation, the device supplying (or transmitting) power will be referred to as the P-Source device, and the device receiving (or supplying) power will be referred to as the P-Sync device. Additionally, a device that simultaneously supports the functions of both the P-Source device and the P-Sync device will be referred to as a Dual device.

[0067]

[0068] FIG. 2 is a block diagram showing an example of the structure of a Source device and a Sink device according to various embodiments of the present disclosure.

[0069] A detailed description of each component shown in Fig. 2 is as follows.

[0070] (1) Source Device

[0071] A device that sends request messages issuing commands to a Sink Device or receives and processes request messages from a Sink Device.

[0072] A device that supports a UI that processes a response message received from a Sink Device after sending the above request message and delivers it to the user, and a display device that displays the UI, and a device that supports a user input interface that receives user actions through the UI.

[0073] A device that supports a display device for providing a UI that receives, processes, and transmits a request message from the above-mentioned Sink Device to the user, and a device that supports a user input interface that receives user actions through the UI.

[0074] A physical device that transmits or streams content stored in the Source Device's Content Storage to the Sink Device.

[0075] (2) Sink Device

[0076] A device that sends request messages issuing commands to a Source Device or receives and processes request messages from a Source Device.

[0077] A device that supports a UI that processes a response message received from a Source Device after sending the above request message and delivers it to the user, and a display device that displays the UI, and a device that supports a user input interface that receives user actions through the UI.

[0078] A device that supports a UI that receives and processes a request message from the above-mentioned Source Device, delivers it to the user, and displays it, and a device that supports a user input interface that receives user actions through the UI.

[0079] A physical device that receives content from a source device or streams it to provide content rendering to the user.

[0080] (3) Network interface

[0081] A physical device that enables the transmission of messages or data, such as commands, requests, actions, and responses, between devices.

[0082] (4) Memory unit

[0083] As an optional device implemented in various types of devices, a volatile physical device (e.g., Memory) in which various types of data are temporarily stored

[0084] (5) Control unit

[0085] Overall operation control of Source and Sink Devices

[0086] (6) Display

[0087] Data received through the network interface or data stored in the Content Storage is displayed on the screen under the control of the Control Unit.

[0088] (7) Multimedia module

[0089] Device for playing various types of multimedia

[0090] The multimedia module can be implemented within the control unit or separately from the control unit.

[0091] (8) Storage

[0092] A non-volatile physical device capable of storing various types of data (e.g., SD card)

[0093] (9) Power Supply

[0094] A device that receives external and internal power under the control of a control unit and supplies the power necessary for the operation of each component.

[0095] (10) EDID / Display block (EEPROM, memory)

[0096] EEPROM storing EDID information

[0097] (11) Video Encoder

[0098] A device that compresses video to be transmitted via HDMI / DisplayPort Tx

[0099] (12) Video Decoder

[0100] A device that decompresses compressed video received via HDMI / DisplayPort Rx

[0101]

[0102] FIG. 3 is a block diagram showing an example of the structure of a Source device and a Sink device according to various embodiments of the present disclosure.

[0103] A detailed description of each component shown in Fig. 3 is as follows.

[0104] (1) Source Device

[0105] A device that sends request messages issuing commands to a Sink Device or receives and processes request messages from a Sink Device.

[0106] A device that supports a UI that processes a response message received from a Sink Device after sending the above request message and delivers it to the user, and a display device that displays the UI, and a device that supports a user input interface that receives user actions through the UI.

[0107] A device that supports a display device for providing a UI that receives, processes, and transmits a request message from the above-mentioned Sink Device to the user, and a device that supports a user input interface that receives user actions through the UI.

[0108] A physical device that transmits or streams content stored in the Source Device's Content Storage to the Sink Device.

[0109] (2) Sink Device

[0110] A device that sends request messages issuing commands to a Source Device or receives and processes request messages from a Source Device.

[0111] A device that supports a UI that processes a response message received from a Source Device after sending the above request message and delivers it to the user, and a display device that displays the UI, and a device that supports a user input interface that receives user actions through the UI.

[0112] A device that supports a UI that receives and processes a request message from the above-mentioned Source Device, delivers it to the user, and displays it, and a device that supports a user input interface that receives user actions through the UI.

[0113] A physical device that receives content from a source device or streams it to provide content rendering to the user.

[0114] (3) Network interface

[0115] A physical device that enables the transmission of messages or data, such as commands, requests, actions, and responses, between devices.

[0116] (4) Memory unit

[0117] As an optional device implemented in various types of devices, a volatile physical device (e.g., Memory) in which various types of data are temporarily stored

[0118] (5) Control unit

[0119] Overall operation control of Source and Sink Devices

[0120] (6) Display

[0121] Data received through the network interface or data stored in the Content Storage is displayed on the screen under the control of the Control Unit.

[0122] (7) Multimedia module

[0123] Device for playing various types of multimedia

[0124] The multimedia module can be implemented within the control unit or separately from the control unit.

[0125] (8) Storage

[0126] A non-volatile physical device capable of storing various types of data (e.g., SD card)

[0127] (9) Power Supply

[0128] A device that receives external and internal power under the control of a control unit and supplies the power necessary for the operation of each component.

[0129] (10) EDID / Display block (EEPROM, memory)

[0130] EEPROM storing EDID information

[0131] (11) Video Encoder

[0132] A device that compresses video to be transmitted via HDMI / DisplayPort Tx

[0133] (12) Video Decoder

[0134] A device that decompresses compressed video received via HDMI / DisplayPort Rx

[0135]

[0136] 1. EDID (Extended Display Identification Data)

[0137] EDID is a type of data structure containing various information about a display device defined by VESA, and it is transmitted to the source device.

[0138] EDID is compatible with the upper 128 bytes from Version 1.0 to 1.4, and from version 1.3 onwards it is called Enhanced EDID, and an EDID extension block is added after the upper 128 bytes to insert additional data.

[0139] EDID was deprecated up to Version 1.2, and Version 1.3 is widely used in IT display devices, CE display devices, and video interfaces (e.g., HDMI).

[0140] The following Table 1 shows the configuration of the EDID.

[0141] Address No. Bytes Description 00h ~ 07h 8 Header information. Fixed as 00 FF FF FF FF FF FF 00. 08h ~ 11h 10 Vendor / Product Identification. Manufacturer, product code, serial number, and manufacturing date 12h ~ 13h 2 EDID Structure Version / Revision 14h ~ 18h 5 Basic Display Parameters / Features. Video Input definition (analog or digital), Max. Horizontal Image Size, Max. Vertical Image Size, Display Transfer Characteristic (Gamma), Feature Support (Standby, Suspend, Display Type, Standard Default Color space (sRGB), Preferred Timing Mode support, etc.) 19h ~ 22h 10 Color Characteristics. Information related to color and white point. Displayed as xy coordinates of Red, Green, Blue, and White in color space. 23h ~ 25h 3 Established Timings. Describes commonly used timing modes. 26h ~ 35h 16 Standard Timings. Eight standard timing descriptors are described, with each descriptor containing information on the range of horizontal active pixels, Image Aspect Ratio, and Refresh Rate (60 ~ 123Hz). Timing not included in Established Timing is described in accordance with the VESA DMT standard or using timing information calculated via GTF. 36h ~ 7 Dh 7 2 Detailed Timing Descriptors.Detailed timing information for the resolutions supported by the display is described, and there are four descriptors. The first descriptor is Preferred Detailed timing. The second descriptor indicates secondary detailed timing or additional monitor information (Serial Number, Range Limits, Name), and the remaining two descriptors contain additional monitor information. Monitor Range Limit and Name must be specified. 7Eh1Extension Flag. Specifies the number of additional EDID extension blocks. 7Fh1Checksum.

[0142] 2. CEA 861 EDID Extension Block

[0143] The timing information described in the EDID is for IT display devices, and the EDID 1.3 Extension Block from CEA 861 is used to represent the timing information of CE display devices.

[0144] Version 3 CEA Extension is defined in the CEA 861B standard and specifies four optional Data Blocks (Video, Audio, Speaker Allocation, Vendor Specific).

[0145]

[0146] Byte #0Tag. 0x02 1Revision Number. 0x03 2Byte number where the 18-byte Detailed Timing Descriptor (DTD) starts Offset d value 3Indication of underscan, audio support, YCBCR 4:4:4 or YCBCR 4:2:2 support, number of supported native DTDs 4Start of data block collectiond -1End of data block collectiond Start of 18-byte DTD. Follows EDID DTD format. d+(18*n)-1End of 18-byte DTD. n is the number of descriptors included +(18*n)Beginning of Padding. 0x0012 6End of Padding. 0x0012 7Checksum

[0147] The following Table 3 shows the Video Data Block.

[0148] Byte #Bits 5-7Bits 0-40Video Tag CodeShort Video Descriptor Total number of bytes (L1)1CEA Short Video Descriptor 1L1CEA Short Video Descriptor L1

[0149] The Short Video Descriptor contains the Video Identification Code defined in CEA-861.

[0150]

[0151] The following Table 4 shows the Audio Data Block.

[0152] Byte #Bits 5-7Bits 0-40Audio Tag CodeShort Audio Descriptor Total number of bytes (L2)1 ~ 3CEA Short Audio Descriptor 14 ~ 3*L2CEA Short Video Descriptor L2 / 3

[0153] The Short Audio Descriptor contains the Audio Format Code defined in CEA-861.

[0154]

[0155] The following Table 5 shows the Speaker Allocation Data Block.

[0156] Byte #Bits 5-7Bits 0-40Speaker allocation Tag CodeSpeaker Allocation Total number of bytes (L3=3)1 ~ 3Speaker Allocation Data Block Payload

[0157] The Speaker Allocation Data Block Descriptor contains the Data Block Payload defined in CEA-861.

[0158]

[0159] 3. Vendor-Specific Data Block

[0160] As a block where vendor-specific data can be defined, HDMI uses this data block to define HDMI-specific data. The data block below is HF-VSDB.

[0161] Byte / Bits #765432100Vendor Specific Tag Code (=3)Length (=N)1IEEE OUI, Third Octet (0xD8)2IEEE OUI, Second Octet (0x5D)3IEEE OUI, First Octet (0xC4)4Version (=1)5Max_TMDS_Character_Rate6SCDC_PresentRR_CapableRsvd(0)Rsvd(0)LTE_340Mcsc_scrambleIndependent_viewDual_ view3D_OSD_Disparity7Rsvd(0)Rsvd(0)Rsvd(0)Rsvd(0)Rsvd(0)DC_48bit_420DC_36bit_420DC_30bit_420...NReserved(0)

[0162] (1) Length : The total length of the data block, with a minimum value of 7 and a maximum value of 31.

[0163] (2) IEEE OUI: As an IEEE Organizationally Unique Identifier, the OUI assigned to the HDMI Forum is 0xC45DD8.

[0164] (3) Version: The version number of HF-VSDB (HDMI Forum-VSDB) is 1.

[0165] (4) Max_TMDS_Character_Rate : Indicates the maximum supported TMDS Character Rate. If the Sink device does not support 340 MaxDS or higher, set it to 0, and if it does support it, set it to 1.

[0166] (5) 3D_OSD_Disparity: When set to 1, it indicates that the Sink device supports receiving 3D_OSD_Disparity Indication.

[0167] (6) Dual_view : When set to 1, it indicates that the Sink device supports Dual_view signaling reception.

[0168] (6) Independent_view : When set to 1, it indicates that the Sink device supports receiving 3D independent view signaling.

[0169] (7) LTE_340Mcsc_scramble: When set to 1, it indicates that the Sink device supports scrambling at a TMDS character rate of 340Mcss or less. Also, if SCDC_Present is set to 0, this flag must also be set to 0.

[0170] (8) RR_Capable: When set to 1, it indicates that the Sink device can initiate an SCDC read request. And if SCDC_Present is set to 0, this flag must also be set to 0.

[0171] (9) SCDC_Present: When set to 1, it indicates that the Sink supports SCDC functionality.

[0172] (10) DC_48bit_420, DC_36bit_420, DC_30bit_420 : When set to 1, it indicates that Deep Color 4:2:0 pixel encoding is supported at 10 bits / 12 bits / 16 bits per component.

[0173]

[0174] 4. DisplayPort Supported Resolutions

[0175] Unlike HDMI, DisplayPort does not have a defined default mandatory resolution.

[0176] Definition Formats Field Rate Aspect Ratio Support by Standard Technology DisplayPort 1.2a 1.3 SD (Standard Definition) Supports all ED (Enhanced Definition) Supports all HD (High Definition) Supports all Full HD Supports all 4K UHD (Ultra High Definition) 3840x2160p 23.98 / 24Hz 16:9 Support Support3840x2160p25Hz3840x2160p29.97 / 30Hz3840x2160p50Hz3840x2160p59.94 / 60Hz3840x2160p23.98 / 24Hz64:273840x2160p25Hz3840x2160p29.97 / 30Hz3 840x2160p50Hz3840x2160p59.94 / 60Hz4096x2160p23.98 / 24Hz256:1354096x 2160p25Hz4096x2160p29.97 / 30Hz4096x2160p50Hz4096x2160p59.94 / 60Hz8K UHD 7680x4320p 23.98 / 24Hz 16:9 Not supported Not supported (Scheduled for support in 1.3a)

[0177] 5. Display ID Extension Block

[0178] (1) Display ID is a VESA standard created to replace the E-EDID standard and EDID v1.4.

[0179] (2) Display ID v1.1 was published in March '09, and v1.3 was published in September '13.

[0180] (3) This standard has a variety of structures, including new extension formats for embedded displays and 3D displays as well as existing EDID extension formats.

[0181] (4) The Display ID format includes several blocks that describe information about the display, such as video interfaces, display device technology, timing details, and manufacturer information.

[0182] (5) Blocks are flexible up to the length of the header, and the length of each field is variable and the specific number of bytes is not fixed. Only a few data blocks are fixed.

[0183]

[0184] Table 8 shows the Display ID Structure.

[0185] AddressValueDescription00h12hDisplay ID Structure Version 1, Revision 201h00h -> FBhBytes in Section02h ~ 03h00h -> FFhDisplay Product Type Identifier & Extension Count04hBLOCKFirst Data block......(N-2)hBLOCKLast byte of Last valid data block(N-1)h00h -> FFhChecksum

[0186] Table 9 shows the Data Block Format.

[0187] OffsetValueDescription00h00h -> FFhData Block Identification01h0 -> 7Block Revision and other data02h00h -> F8hNumber of Payload Bytes 0 -> 24803hDESCRIPTOR1st Data Payload Byte04h.2nd Data Payload Byte... (if present)...

[0188] (1) Data Block Identification: Displays the tag of each data block.

[0189] (2) Block revision and other data: The revision increases whenever a bit is added or changed in the block.

[0190] (3) Number of Payload Bytes: Indicates the number of bytes of payload used in a single data block.

[0191] (4) 1 ~ N Data Payload Byte : Starting from 03h, it explains what role each Data Payload Byte plays.

[0192]

[0193] 6. DDC (Display Data Channel)

[0194] (1) Protocol standard for transmitting digital information between a monitor and a computer's graphics adapter defined by VESA (Video Electronics Standard Association)

[0195] (2) The monitor transmits information on supported display modes to the graphics adapter, and the graphics adapter transmits video to the monitor accordingly.

[0196] (3) Before the DDC standard was established, the VGA standard used four pins (Pin 11, 12, 4, 15) of the analog VGA connector to recognize monitor types, and only Pins 11, 12, and 4 were used, and 7 types of monitor types could be recognized.

[0197] (4) DDC version 1 (established in 1994)

[0198] (4-1) Defines EDID (Extended Display Identification Data), a binary file format that describes monitor information.

[0199] (4-2) Use Pin 12 as the data line and continuously transmit 128-byte EDID blocks from the monitor to the computer.

[0200] (5) DDC version 2 (established in 1996)

[0201] (5-1) EDID is not defined in DDC but is defined as an independent standard as a companion standard.

[0202] (5-2) Defined based on the I2C Serial Bus, Pin 12 is used as the data line of the I2C Bus, and Pin 15 is used as the clock line of the I2C Bus.

[0203] (5-3) Pin 9: Used to supply 5V DC power (up to 50mA) from the computer to the monitor to read the EDID stored in the EEPROM even when the monitor power is off.

[0204] (5-4) The monitor is assigned the 7-bit I2C address 50h as a slave device of the I2C Bus.

[0205] (5-5) Allows EDID storage capacity of up to 28 bytes = 256 bytes with an 8-bit data offset.

[0206] (6) E-DDC

[0207] (6-1) Version 1 was established in 1999 as a standard replacing DDC versions 1 and 2, and allows up to 32 Kbytes of display information storage capacity for use with Enhanced EDID.

[0208] (6-2) By applying a new I2C addressing scheme using 8-bit segment indices (0x00 to 0x7F), 128 segments (1 segment = 256 bytes) can be accessed, allowing access up to 32K bytes.

[0209] (6-3) E-DDC version 1.1 was established in 2004 and includes support for CE devices and video interfaces other than VGA (e.g., HDMI).

[0210] (6-4) E-DDC version 1.2 was established in 2007 and includes support for DisplayPort and DisplayID.

[0211]

[0212] 7. SCDCS (Status and Control Data Channel Structure)

[0213] (1) SCDCS is a one-to-one communication protocol based on I2C serial communication that enables data exchange between HDMI source devices and sink devices.

[0214] (2) Extension of the existing I2C standard: A function is added in which the I2C slave Sink device requests a status check read from the I2C master Source device, and the Source device, upon receiving this, reads the corresponding status from the Sink device.

[0215] Table 10 shows an example of a specific configuration of SCDCS.

[0216] OffsetR / WName0x01RSink Version0x02R / WSource Version0x10R / WUpdate_00x11R / WUpdate_10x12-0x1FRReserved for Update Related Uses0x20R / WTMDS_Config0x21RScrambler_Status0x30R / WConfig_00x31-0x3FRReserved for Configuration0x40RStatus_Flag_00x41RStatus_Flag_10x42-0x4FRReserved for Status Related Uses0x50RErr_Det_0_L0x51RErr_Det_0_H0x52RErr_Det_1_L0x53RErr_Det_1_H0x54RErr_Det_2_L0x55RErr_Det_2_H0x56RErr_Det_Checksum0xC0R / WTest_Config_00xC1`0xCFRReserved for test features0xD0RManufacturer IEEE OUI, Third Octet0xD1RManufacturer IEEE OUI, Second Octet0xD2RManufacturer IEEE OUI, First Octet0xD3-0xDDRDevice ID0xDE-0xFFR / WManufacturer SpecificAll Remaining OffsetsRReserved

[0217] The following is a detailed description of each component of the SCDCS.

[0218] (1) Sink Version : Displays the version information of the SCDCS compliant sink device. Set to 1.

[0219] (2) Source Version: SCDCs compliant. When a source device reads an E-EDID from a sink device and the SCDC_Present of the E-EDID is set to 1, the Source Version of the SCDCs is set to 1.

[0220] (3) Update Flags (Update_0, Update_1): When there is a change in the information (Status, Character Error Detect, etc.) that the Sink device must notify the Source device, the corresponding bit is set to 1.

[0221] (4) TMDS Configuration (TMDS_Config): TMDS_Bit_Clock_Ratio and Scrambling_Enable each occupy 1 bit. If the Source device wants to enable the scrambling function of the Sink device, set the bit to 1. If TMDS_Bit_Clock_Ratio is 1 / 10, set it to 0, and if it is 1 / 40, set it to 1.

[0222] (5) Scrambler Status: When the Sink device detects a scrambled control code sequence, the corresponding bit is set to 1.

[0223] (6) Configuration (Config_0): A field for configuring information related to the capabilities of Source and Sink devices. Currently, there is only the RR_Enable field, which indicates whether the Source device supports the Sink device's Read Request.

[0224] (7) Status Flags (Status_Flag_0, Status_Flag_1) : Indicates whether data received through Clock, channel 0, 1, and 2 has been successfully decoded.

[0225]

[0226] FIG. 4 is a diagram showing the initial connection sequence between a source device and a sink device according to the prior art.

[0227] Step 0 (Cable Connection): The user connects a cable between the two devices to transmit audio / video (A / V) data from the source device to the sink device.

[0228] Step 1 (Providing +5V power line): The source device applies a +5V power line to a high level and transmits current to operate the EEPROM and related circuits where the sink device's EDID (Extended Display Identification Data) information is stored.

[0229] Step 2 (HPD provision): The EDID-related circuit of the sink device activated through Step 1 transitions the hot plug detection (HPD) line, which was maintained at a low level, to a high level to inform the source device that the cable is properly connected and EDID information is accessible.

[0230] Step 3 (EDID read request): The source device confirms that the HPD line has transitioned to a high level through Step 2 and requests the sink device to read the EDID (or DPID) information through the display data channel (DDC).

[0231] Step 4 (Transmission of EDID information): In response to the source device's request to read the EDID, the sink device transmits the EDID information stored in the EEPROM to the source device via the DDC.

[0232] After step 4, the source device parses the received EDID information and determines the operating parameters of the A / V data to be transmitted to the sink device, such as timing, format, and whether to use DSC compression.

[0233] Step 5 (Audio and Video Transmission): After determining the parameters, the source device transmits audio and video to the sink device. In the case of the HDMI standard, since the transmission clock is synchronized with the video, the audio must be recovered through a separate clock mechanism. During this recovery process, desynchronization between the audio and video may occur primarily.

[0234] This conventional technology has the problem of lacking separate time information for audio and video transmission. As a result, during playback at the sink device, audio and video are processed and played separately, and since time information to improve this is not transmitted, a synchronization problem between audio and video persists.

[0235]

[0236] Figure 5 is a diagram illustrating the asynchronous problem that occurs during audio and video transmission according to the prior art and the prior art processing method for it.

[0237] As illustrated in Fig. 5, when a source device continuously transmits audio / video data to a sink device, an 'AV Sync Broken' phenomenon occurs at the receiving sink device.

[0238] The conventional approach to this was to handle AV synchronization by applying a delay to the audio at the sink device, as shown at the bottom of the diagram. In other words, sink manufacturers applied an intentional delay value to the audio, which has a relatively small amount of data, in an attempt to align the playback timing with the video data, which has a longer processing pipeline.

[0239] However, there were several problems with this delay-based method. For example, although features such as 'Dynamic Audio LipSync' were defined within the HDMI CEC 2.0 standard, the protocol was not widely used in actual environments. In addition, since the delay value required for synchronization could vary depending on the connected device (source device), it was practically difficult for the sink device to handle synchronization by responding to all different delay values.

[0240]

[0241] FIG. 6 is a diagram showing a sequence for performing audio and video synchronization using time information (Time Stamp) according to one embodiment of the present disclosure.

[0242] Specifically, FIG. 6 is a drawing illustrating the improvement proposal of the present disclosure for solving the asynchronous problem of the prior art.

[0243] As shown in Fig. 6, when the source device transmits audio / video data to the sink device, it generates a common time stamp and sends it along with the data.

[0244] Specifically, the source device generates a 'Time Stamp for Audio' and a 'Time Stamp for Video', adds (inserts) them into the audio packet and video packet, respectively, and transmits them.

[0245] Subsequently, the sink device performs synchronization by referencing audio and video data based on this received time stamp information. As a result, since the sink device can accurately perform synchronization by utilizing time information generated from the original playback device (source), the synchronized audio and video can be enjoyed in a 'no issue' state without the synchronization breakdown phenomenon seen in conventional technology.

[0246]

[0247] FIG. 7 is a diagram schematically showing the structure of a video frame to which an embodiment of the present disclosure can be applied.

[0248] According to the 'Video Frame Configuration' illustrated in Fig. 7, a video frame is configured based on vertical synchronization (VSYNC) and horizontal synchronization (HSYNC) signals.

[0249] 1. Horizontal composition (horizontal): A line consists of a total of 858 pixels. This includes 720 active pixels corresponding to the 'Active Video' area displayed on the actual screen and 138 pixels corresponding to the 'horizontal blanking' section for signal transmission.

[0250] 2. Vertical composition (vertical): One frame consists of a total of 525 lines. This includes 480 active lines corresponding to the 'Active Video' area and 45 lines corresponding to the 'vertical blanking' section.

[0251] In addition, when transmitting TMDS (Transition Minimized Differential Signaling), this frame is divided into three different periods (TMDS Periods).

[0252] 1. Video Data Period: This corresponds to the 'Active Video' area and transmits actual pixel data. This leads to the 'Video' block on the right side of the drawing.

[0253] 2. Data Island Period: Located mainly in horizontal and vertical blanking sections, it is used to transmit audio packets or other auxiliary data. This leads to the 'Audio' block on the right side of the diagram.

[0254] 3. Control Period: Used to transmit synchronization and control signals such as HSYNC and VSYNC.

[0255] The present disclosure is characterized by performing synchronization by inserting a common time information (Time Stamp) into video data transmitted through a 'video data segment' and audio data transmitted through a 'data island segment' in such a frame structure.

[0256]

[0257] FIG. 8 is a block diagram illustrating the processing process of video source data in one embodiment of the present disclosure.

[0258] As illustrated in Fig. 8, the processing begins with a 'Video Source'. As specified in the annotation, this refers to a source that 'actually includes both audio and video'.

[0259] Source data can choose one of the following two paths:

[0260] 1. Data is transmitted directly to the 'Unencoded HDMI (3 Channel + Clock)' block without compression.

[0261] 2. After the data is compressed through the 'VESA DSC Encoder', it is transferred to the 'unencoded HDMI' block.

[0262] The data can then undergo an 'Encryption (Optional)' step. This encrypted 'Tri-Bytes' output branches into the two main transmission pipelines of the HDMI standard.

[0263] 1. TMDS Path: Data is processed through the 'TMDS Scrambler (Section 6.1.2) and Encoder (H14b Section 5)'. This is the traditional method used in HDMI 2.0 and earlier versions.

[0264] 2. FRL Path: Data is processed through the FRL (Fixed Rate Link) pipeline. This corresponds to the HDMI 2.1 standard, which supports higher bandwidth.

[0265] (1) First, data is packetized in the 'FRL Packetizer'.

[0266] (2) Next, the 'FRL Character Block and Super Block Mapping' step is performed.

[0267] (3) Afterwards, in the ‘RS FEC Parity Generation and Insertion’ step, a parity bit for error correction is added.

[0268] (4) Finally, the final transmission signal is generated by passing through the ‘FRL scrambler (Section 6.5.7) and encoder (Section 6.5.8)’.

[0269]

[0270] FIG. 9 is a diagram illustrating the Active Video period and the Data Island Period within a video frame according to one embodiment of the present disclosure.

[0271] Specifically, FIG. 9 is a drawing for illustrating a method of inserting common time information into video and audio according to one embodiment of the present disclosure.

[0272] Figure 9 shows that, similar to Figure 7, a video frame is divided into a 'Video Data Period' where 'Active Video' is transmitted and a 'Data Island Period' where 'Audio' packets, etc. are transmitted. In the present disclosure, the method for synchronizing video and audio by adding timing information is as follows.

[0273] 1. Video Side:

[0274] (1) Video transmitted through the ‘video data section’ has the corresponding frame number (Video Frame Number) added.

[0275] (2) The frame number and time stamp information inserted at this time must be the same as the values ​​inserted into the audio.

[0276] (3) Since an error of about 1 to 2 frames in this time information is not critical to synchronization, it can be inserted at least once every 2 frames.

[0277] 2. Audio Side:

[0278] (1) When a video frame number is inserted, the same value (e.g., frame number and counter value) that can be synchronized with the audio is inserted into the 'Data Island Packet' that transmits audio information.

[0279] (2) That is, since audio is additional data for the corresponding video frame, the same frame value is inserted through a separate packet.

[0280]

[0281] The table for Data Island Packet Types defines the 'Packet Type' based on the 'Packet Type Value'. For example, a packet type value of 0x01 represents 'Audio Clock Regeneration (N / CTS)', 0x02 represents 'Audio Sample (L-PCM and IEC 61937 compressed formats)', and values ​​of 0x80 or higher represent the InfoFrame Packet type.

[0282] In this disclosure, for audio and video synchronization, it is proposed to use a newly defined packet type value that is not previously defined, for example, '0x1F', as an 'Audio Time stamp' packet.

[0283] A single frame consists of an 'Active Video' area containing actual video data and an 'Audio Packet and audio' area where audio packets are transmitted. These audio packets are typically transmitted through a 'Data Island Period,' which corresponds to the video's blanking period.

[0284] The newly defined '0x1F : Audio Time stamp' packet is included in this 'Audio Packet and audio' area and transmitted from the source device to the sink device. This packet contains a synchronization value identical to the time information (e.g., frame number and counter value) of the corresponding 'Active Video' frame.

[0285] In the Sink, when the relevant information arrives, it synchronizes by referencing the value in the Frame.

[0286]

[0287] [Explanation regarding Source device claim]

[0288] The embodiments described above will be explained in detail below with reference to FIG. 10 regarding the operation of the terminal. The methods described below are distinguished only for the convenience of explanation, and it is obvious that as long as they are not mutually excluded, a part of one method may be substituted with a part of another method or combined with one another and applied.

[0289] FIG. 10 is a drawing illustrating an example of the operation process of a Source device according to various embodiments of the present disclosure.

[0290] According to various embodiments of the present disclosure, a method performed by a Source device is provided.

[0291] The Source device includes a processor; memory; and a transceiver. The memory stores instructions for performing operations based on execution by the processor.

[0292] In step S1001, the Source device generates common time information for video data and audio data.

[0293] In step S1002, the Source device inserts the time information into the video data.

[0294] In step S1003, the Source device inserts the time information into the audio data.

[0295] In step S1004, the Source device transmits the video data containing the time information and the audio data containing the time information to the Sink device.

[0296]

[0297] According to various embodiments of the present disclosure, the time information may include a frame number and a counter value.

[0298] According to various embodiments of the present disclosure, the step S1002 of including the time information in the video data may include the step of inserting the time information into a video frame.

[0299] According to various embodiments of the present disclosure, the time information may be inserted every at least two video frames.

[0300] According to various embodiments of the present disclosure, the step S1003 of including the time information in the audio data may include the step of inserting the time information into a Data Island Packet that transmits the audio data.

[0301] According to various embodiments of the present disclosure, the time information may be defined and inserted as an audio time stamp packet type.

[0302] According to various embodiments of the present disclosure, the video data and the audio data may be transmitted via an HDMI (High-Definition Multimedia Interface) or DisplayPort interface.

[0303]

[0304] According to various embodiments of the present disclosure, a Source device is provided. The Source device comprises a processor; a memory; and a transceiver, and the processor may be configured to perform the method of operation of the Source device according to FIG. 10.

[0305] According to various embodiments of the present disclosure, a device for controlling a Source device is provided. The device comprises at least one processor and at least one memory operably connected to the at least one processor. The at least one memory may be configured to store instructions for performing a method of operating the Source device according to FIG. 10 based on execution by the at least one processor.

[0306] According to various embodiments of the present disclosure, one or more non-transitory computer-readable media (CRMs) storing one or more instructions are provided. The one or more instructions perform operations based on execution by one or more processors, and the operations may include a method of operation of a Source device according to FIG. 10.

[0307]

[0308] [Explanation regarding Sink device claim]

[0309] The embodiments described above will be explained in detail below with reference to FIG. 9 regarding the operation of the terminal. The methods described below are distinguished only for the convenience of explanation, and it is obvious that, as long as they are not mutually excluded, a part of one method may be substituted with a part of another method or combined with one another and applied.

[0310] FIG. 11 is a drawing illustrating an example of the operation process of a Sink device according to various embodiments of the present disclosure.

[0311] According to various embodiments of the present disclosure, a method performed by a sink device is provided.

[0312] The Sink device includes a processor; memory; and a transceiver. The memory stores instructions for performing operations based on execution by the processor.

[0313] In step S1101, the Sink device receives video data containing first time information and audio data containing second time information from the Source device.

[0314] In step S1102, the Sink device obtains the first time information from the video data.

[0315] In step S1103, the Sink device obtains the second time information from the audio data.

[0316] In step S1104, the Sink device compares the first time information and the second time information to synchronize the video data and the audio data.

[0317]

[0318] According to various embodiments of the present disclosure, the first time information and the second time information may include a frame number and a counter value.

[0319] According to various embodiments of the present disclosure, the first time information may be included in a video frame in which the video data is transmitted.

[0320] According to various embodiments of the present disclosure, the first time information can be obtained every at least two video frames.

[0321] According to various embodiments of the present disclosure, the second time information may be included in a Data Island Packet in which the audio data is transmitted.

[0322] According to various embodiments of the present disclosure, the second time information can be obtained from a packet defined as an audio time stamp packet type.

[0323] According to various embodiments of the present disclosure, the video data and the audio data may be received via an HDMI (High-Definition Multimedia Interface) or DisplayPort interface.

[0324]

[0325] According to various embodiments of the present disclosure, a sink device is provided. The sink device comprises a processor; a memory; and a transceiver, and the processor may be configured to perform the method of operation of the sink device according to FIG. 11.

[0326] According to various embodiments of the present disclosure, a device for controlling a sink device is provided. The device comprises at least one processor and at least one memory operably connected to the at least one processor. The at least one memory may be configured to store instructions for performing a method of operating a sink device according to FIG. 11 based on execution by the at least one processor.

[0327] According to various embodiments of the present disclosure, one or more non-transitory computer-readable media (CRMs) storing one or more instructions are provided. The one or more instructions perform operations based on execution by one or more processors, and the operations may include a method of operation of a Sink device according to FIG. 11.

[0328]

[0329] The claims described in various embodiments of the present disclosure may be combined in various ways. For example, the technical features of the method claims of various embodiments of the present disclosure may be combined to be implemented as a device, and the technical features of the device claims of various embodiments of the present disclosure may be combined to be implemented as a method. Furthermore, the technical features of the method claims and the technical features of the device claims of various embodiments of the present disclosure may be combined to be implemented as a device, and the technical features of the method claims and the technical features of the device claims of various embodiments of the present disclosure may be combined to be implemented as a method.

Claims

1. In the method of operating a source device, A step of generating common time information for video data and audio data; A step of inserting the above time information into the above video data; The step of inserting the above time information into the above audio data; and A step comprising transmitting the video data including the time information and the audio data including the time information to a sink device. method.

2. In Paragraph 1, The above time information includes a frame number and a counter value, method.

3. In Paragraph 1, The step of including the above time information in the above video data includes the step of inserting the above time information into a video frame. method.

4. In Paragraph 3, The above time information is inserted every at least two video frames, method.

5. In Paragraph 1, The step of including the time information in the audio data comprises the step of inserting the time information into a Data Island Packet that transmits the audio data. method.

6. In Paragraph 5, The above time information is defined as an Audio Time Stamp packet type and inserted, method.

7. In Paragraph 1, The above video data and the above audio data are transmitted via an HDMI (High-Definition Multimedia Interface) or DisplayPort interface, method.

8. In the method of operating a sink device, A step of receiving video data containing first time information and audio data containing second time information from a source device; A step of obtaining the first time information from the above video data; A step of obtaining the second time information from the above audio data; and A method comprising the step of synchronizing the video data and the audio data by comparing the first time information and the second time information. method.

9. In Paragraph 8, The first time information and the second time information above include a frame number and a counter value, method.

10. In Paragraph 8, The above first time information is included in the video frame in which the video data is transmitted, method.

11. In Paragraph 10, The above first time information is acquired every at least two video frames, method.

12. In Paragraph 8, The above second time information is included in the Data Island Packet through which the audio data is transmitted, method.

13. In Paragraph 12, The above second time information is obtained from a packet defined as an audio time stamp packet type, method.

14. In Paragraph 8, The above video data and the above audio data are received via an HDMI (High-Definition Multimedia Interface) or DisplayPort interface, method.

15. In a source device, It includes a processor; memory; and a transceiver, The above memory stores instructions for performing operations based on execution by the processor, and The above operations are, A step of generating common time information for video data and audio data; A step of inserting the above time information into the above video data; The step of inserting the above time information into the above audio data; and A step comprising transmitting the video data including the time information and the audio data including the time information to a sink device. Source device.

16. In Paragraph 15, The above time information includes a frame number and a counter value, Source device.

17. In Paragraph 15, The step of including the above time information in the above video data includes the step of inserting the above time information into a video frame. Source device.

18. In Paragraph 17, The above time information is inserted every at least two video frames, Source device.

19. In Paragraph 15, The step of including the time information in the audio data comprises the step of inserting the time information into a Data Island Packet that transmits the audio data. Source device.

20. In Paragraph 19, The above time information is defined as an Audio Time Stamp packet type and inserted, Source device.

Images