Provenance recovery client

Abstract

Systems and methods for providing provenance authentication of content. A provenance authentication process analyzes audiovisual media of unknown origin an seeks to discover and authenticate related provenance claims using C2PA metadata and open watermarking standards. Disclosed embodiments disclose details of a provenance recovery client that facilitates the provenance authentication process through various functions including the configuration and reporting of soft-binding assertions.

Description

PROVENANCE RECOVERY CLIENT FIELD OF INVENTION

[0001] The present invention generally relates to provenance determination and authentication of content.BACKGROUND

[0002] This section is intended to provide a background or context to the disclosed embodiments that are recited in the claims. The description herein may include concepts that could be pursued but are not necessarily ones that have been previously conceived or pursued. Therefore, unless otherwise indicated herein, what is described in this section is not prior art to the description and claims in this application and is not admitted to be prior art by inclusion in this section.

[0003] Linear broadcast streams are subject a variety of distribution routes and various encoding and transcoding processes. This presents challenges when attempting to authenticate and determine the provenance of the broadcast content downstream of these activities.BRIEF DESCRIPTION OF THE DRAWINGS

[0004] FIG. 1 is a block diagram of metadata and watermarking architecture showing the relationship between the registered content distributors, media objects, their embedded watermarks, associated cryptographic metadata, and the canonical representations of the media object itself in accordance with an exemplary embodiment.

[0005] FIG. 2 illustrates an exemplary production flow in which watermarks are applied to enable provenance across multiple distribution paths, with asset watermarks applied to pre-recorded assets and service watermarking continuously applied to a linear playout stream in accordance with an exemplary embodiment.

[0006] FIG. 3 illustrates a media object validation scenario in accordance with an exemplary embodiment.

[0007] FIG. 4 illustrates a media object canonical processing scenario in accordance with an exemplary embodiment.

[0008] FIG. 5 shows data hash segments, cryptographic metadata, and watermarks in the scenario of a live news broadcast with a watermark consisting of a constant service / asset identifier component and a time-varying index code in accordance with an exemplary embodiment.

[0009] FIG. 6 shows the creation of a canonical media object where the watermark in a live recording is time-varying in accordance with an exemplary embodiment.

[0010] FIG. 7 illustrates a C2PA manifest in accordance with an exemplary embodiment.

[0011] FIG. 8 illustrates an OSI abstraction model of the architecture of the ATSC watermarking system in accordance with an exemplary embodiment.

[0012] FIG. 9 illustrates a typical use case of an open metadata retrieval architecture in accordance with an exemplary embodiment.

[0013] FIG. 10 illustrates an architectural diagram of a representative provenance authentication workflow that employs provenance recovery in accordance with an exemplary embodiment.

[0014] FIG. 11 illustrates a block diagram of a device that can be used for implementing various disclosed embodiments.SUMMARY OF THE INVENTION

[0015] This section is intended to provide a summary of certain exemplary embodiments and is not intended to limit the scope of the embodiments that are disclosed in this application.

[0016] Systems and methods for recovering provenance information in content. The present disclosure incudes a method for providing provenance information for media including receiving input media having components with ATSC soft binding watermarks embedded therein and detecting and processing the watermarks using watermark detectors, building a Watermark Segment List using the output of the watermark detectors, creating a Watermark Segment Record, establishing the validity of each Watermark Segment Record, building a Provenance Recovery Record List, including Provenance Recoveiy Records which each provide a description of a clip of one or more time-aligned component taken from a media asset by converting Watermark Segment Records into Provenance Recovery records, and outputting the Provenance Recovery List and Watermark Segment List, to a provenance authentication process.

[0017] The present disclosure includes a system for providing provenance authentication of media content including a provenance authentication unit for receiving and analyzing media content to discover and authenticate related provenance claims using C2PA metadata and generating an output including, when possible, authenticated content and trust signals. An operating system services unit provides access to platform resources including dynamic memory management, persistent storage, and network servers. A C2PA validator is used by the provenance authentication unit to locate manifests in the media, validate those manifests, and use hard bindings in valid manifests to authenticate input media. A media decoder converts media from media transport and encoding formats into demultiplexed and decoded media components. A watermark detector for detecting watermarks in audio or video components of the media, and a recovery client uses the watermark detector to detecting watermark segments in the media, uses the operating system services unit to retrieve metadata and manifests associated with the watermark segments from network servers and store them locally, anduses the C2PA validator to validate retrieved manifests, wherein the details of valid soft- binding assertions are established and made available to the provenance authentication unit.

[0018] These and other advantages and features of disclosed embodiments, together with the organization and manner of operation thereof, will become apparent from the following detailed description when taken in conjunction with the accompanying drawings.DETAILED DESCRIPTION OF CERTAIN EMBODIMENTS

[0019] In the following description, for purposes of explanation and not limitation, details and descriptions are set forth in order to provide a thorough understanding of the disclosed embodiments. However, it will be apparent to those skilled in the art that the present invention may be practiced in other embodiments that depart from these details and descriptions.

[0020] Additionally, in the subject description, the word “exemplary” is used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as “exemplary” is not necessarily to be construed as referred or advantageous over other embodiments or designs. Rather, use of the word exemplary is intended to present concepts in a concrete manner.

[0021] Embodiments are disclosed which enable end-to-end interoperable provenance authentication using watermarks and C2PA metadata.

[0022] Interoperable Provenance Authentication of Broadcast Media using Open Standards-based Metadata, Watermarking and Cryptography

[0023] The spread of false and misleading information is receiving significant attention from legislative and regulatory bodies. Consumers place trust in specific sources of information, so a scalable, interoperable method for determining the provenance and authenticity of information is needed. The disclosed embodiments address the posting ofbroadcast news content to a social media platform, the role of open standards, the interplay of cryptographic metadata and watermarks when validating provenance, and likely success and failure scenarios. The open standards for cryptographically authenticated metadata developed by the Coalition for Provenance and Authenticity (C2PA) and for audio and video watermarking developed by the Advanced Television Systems Committee (ATSC) are well suited to address broadcast provenance. The disclosed embodiments teach methods for using these standards for optimal success.

[0024] Introduction

[0025] In our interconnected world, information flows ceaselessly, shaping opinions, policies, and societies. Within this digital torrent false and misleading information often obscures the truth.

[0026] False information may take the form of misinformation, spread when well- intentioned individuals share what they found online, neglecting to verify what they found. Or it may be disinformation, false or misleading information intentionally created and spread to deceive.

[0027] Both forms of false information are harmful, and both thrive in the global digital ecosystem. Social media platforms amplify their reach, turning falsehoods into viral storms. A rumor, a manipulated video, a fabricated statistic — these can cascade across screens, eroding public discourse.

[0028] Provenance and Authenticity

[0029] Any attempt to address false information on the web must proceed from an understanding of how people come to place bust in information.

[0030] The prevalence of information ‘bubbles’ demonstrates that people primarily place tiust in specific sources of information. If information appears unaltered and from a trusted source, we often consider that information to be factual.

[0031] In other words, most of us judge what is factual based on the provenance and authenticity of the information, where provenance refers to the origin, history, and chain of custody of a piece of audio-video content, and authenticity refers to whether the content has been manipulated or altered in a way out of the control of the trusted source of the information.

[0032] The Role of Standards

[0033] There are two general methods for conveying provenance and authenticity metadata in association with audio-video content. Metadata can be cryptographically bound to the audio-video content, perhaps stored at the audio-video container level. Metadata can also be embedded as a watermark in the audio-video elementary stream.

[0034] For practical reasons described in herein these two metadata approaches are interdependent. Both cryptographic and watermarking provenance and authenticity methods should provide a reasonable degree of provenance assurance.

[0035] A critical issue to address is the impact of adopting proprietary solutions on interoperability and scalability, an issue often encountered. For example, fifteen years ago, Digital Rights Management (DRM) on the web had not yet been standardized. Prior to the ISO / IEC Common Encryption standard playback devices would have to support every major variety of digital rights management software, and there would be as many versions of the audio-video content as there were DRM systems. Had this continued it would have resulted in a combinatorial explosion, an effective barrier to large scale growth of commercial web media. It is no wonder that Netflix was one of the first companies to recognize the value of the common encryption standard.

[0036] It is reasonable to expect that the same will hold true for provenance and authenticity. For scalability and interoperability, the cryptographic metadata bound to the audio-video container and the watermark metadata embedded in the audio-video elementary streams must include open standard options.

[0037] A solution for provenance and authenticity for broadcast content distributed on social media platforms is described, utilizing metadata, watermarking and cryptographic standards. The disclosed embodiments demonstrate this can be used with broadcast news content while pointing out several important implementation considerations.

[0038] Provenance and Authenticity Success Scenarios

[0039] A provenance and authenticity use case can encompass multiple scenarios, including success scenarios, where everything goes roughly as intended and various exception scenarios, which lead to undesirable outcomes. All these scenarios should be describable as discrete programmatic steps to uncover the functional requirements for addressing provenance and authenticity in practice.

[0040] The following disclosure will examine the details of one specific, provenance and authenticity use case - the posting of what appears to be broadcast news content to a social media platform. What is particularly interesting is the interplay between provenance validation using tamper-evident cryptographic bindings and metadata retrieval using elementary stream watermarks, with a focus on the constituent ‘success’ and ‘exception’ scenarios.

[0041] A broadcaster produces content for linear distribution by an affiliate / network / platfonn operator. This content consists of a series of audio-video programs comprising a single linear broadcast TV channel.

[0042] There are a variety of scenarios where some of the broadcaster content finds its way into Internet distribution and is uploaded to a social media platform. At a minimum it will then be transcoded into a platform’s preferred framerates, resolutions, bitrates, codecs, and container formats. It may also be truncated to meet the platform’s maximum size limits.

[0043] Verifying Authenticity

[0044] Before being posted to a social media platform, broadcast news content may be altered such that there are observable, meaningful differences between what was depicted in the original broadcast and the posted video. This manipulation could be done for artistic, creative, or deceptive reasons, depending on the intention of the editor.

[0045] One way to characterize these differences is to ask whether the posted video is an authentic representation of the original, whether it is time to the original, without any judgement as to whether the original itself depicted what transpired in front of the camera lens and microphone.

[0046] In this definition, an authentic representation of the broadcaster content may not be bit- wise identical to the original, it may be an unaltered clip from the original, or it may be a transcoding of the original, but it may nonetheless accurately represent what was depicted by the original.

[0047] The C2PA standard provides a mechanism for editing the original - e.g., transcoding or clipping - and a means to cryptographically verify the authenticity of the result, but this requires that the tool used to alter the broadcaster content supports the use of this standard. See: https: / / spec.c2pa.Org / specifications / specifications / 2.l / specs / C2PA_Specification.html, which is incorporated herein in its entirety by reference. We believe it is highly unlikely that in the near-term social media platforms will reject content that was edited using a tool that does not implement cryptographic metadata standards.

[0048] Verifying Provenance

[0049] When consuming video in a linear TV receiver, consumers quite reasonably believe that the network / platfomi operator is accurately identifying the channel and content creator.

[0050] Video content on the Internet might misrepresent the identity of the original content creator, the identity of who subsequently transcoded the video and what authorized or unauthorized changes were made.

[0051] One way to characterize this history is to use the term “provenance,” meaning, the identifiable source of the content and an accurate history of the content’s transformation from that source.

[0052] There are cryptographic methods for verifying the provenance of video posted to a social media platform using the C2PA standards, but again it is unlikely in the shortterm that social media platforms will reject videos that do not enable the use of these methods to identify the source of the original content.

[0053] Canonical Representation of a Media Object

[0054] A tamper-evident cryptographic binding to an audio-video media object which contains provenance information can be used to validate the provenance and authenticity of that object. It is surely a successful outcome if the provenance is validated, but what should happen if the provenance and authenticity fail to be verified? What constitutes success in this scenario?

[0055] There are multiple scenarios where the content may have been innocently modified by the user when preparing to post to a social media platform, since even a single bit change to the content will invalidate a cryptographic binding. Generating numerous alerts for innocent alterations to the media could lead to “security alert fatigue,” diminishing user trust in the alert’s salience. Doing nothing is also an unattractive option because it leaves users blind to the “trust signal” conveyed by the presence of a provenance assertion.

[0056] Should the cryptographic verification of the provenance and authenticity of a media object fail a successful outcome is for the social media platform to use an embedded watermark to retrieve the authoritative version - the canonical representationof the media object. This can be done by first using the watermark to retrieve the cryptographic metadata associated with the original media object as distributed and then use that trusted metadata to retrieve the media object’s canonical representation.

[0057] An Approach to Authentication using Metadata and Watermarking

[0058] Architecture

[0059] The provenance and authenticity approach in this paper builds on the relationship between the registered content distributors, media objects, their embedded watermarks, associated cryptographic metadata, and the canonical representations of the media object itself. This relationship is shown in FIG. 1.

[0060] Security Model

[0061] If the tamper evident cryptographic metadata associated with a media object is stored as a component of the media object container, it is relatively easy to remove. A durable embedded watermark can enable cryptographic metadata to be brought back into association with the media object.

[0062] Watermark security is typically maintained by making the watermark difficult to remove or alter by keeping the watermark technology secret. This approach works against availability and interoperability by demanding hardened implementations and strict access controls. It can also provide only weak security assurances because its secrecy impedes comprehensive security assessment. Because recorded broadcast content has a long lifespan on the Internet, the security of “closed” watermarks requires successful long-term protection of the associated secrets. And furthermore, recent advances in attacks on watermarking have demonstrated that advances in artificial intelligence render even robust, secret watermarks automatically removable, further diminishing their potential advantages.

[0063] This motivates a security approach that does not treat the watermark as a root of trust. Instead, we assume that they are durable, i.e. that they survive content processing that causes traditional metadata formats to be lost, but that they are otherwise as mutable as traditional metadata and can be modified or removed by any intermediary. Like traditional metadata, data conveyed via watermarking is treated as untrusted and must be validated using cryptographic methods.

[0064] This same approach was advocated by England et al. in their foundational work on media provenance authentication. See: England, P. et al. 2021. AMP: Authentication of Media via Provenance. 12th ACM Multimedia Systems Conference. July 2021. https: / / doi.org / 10.48550 / arXiv.2001.07886.

[0065] That work, however, assumed the presence of a signature in the watermark payload. We view that signature as unnecessary and assume that the watermark carries only a URL and media timeline. The root of trust is a manifest that has been retrieved using the watermark and cryptographically validated using an appropriate trust list.

[0066] Watermarking Audio-Video Content

[0067] Our success scenario demands a path to validated content regardless of distribution source, which for broadcasters must encompass both linear and on-demand delivery. To achieve this, it must be possible to apply watermarking in asset-based digital publishing as well as within the live production chain.

[0068] FIG. 2 illustrates an exemplary production flow in which watermarks are applied to enable provenance across multiple distribution paths, with asset watermarks applied to pre-recorded assets and service watermarking continuously applied to a linear playout stream.

[0069] The broadcaster may produce content to be published on their website (2-1). They would apply an asset watermark (2-2), generate cryptographic metadata or a“manifest” for that content (2-3), store the asset (2-10) and distribute the asset to their website (2-4).

[0070] The broadcaster may also want to take live and third-party assets (2-5) and prepare them for linear playout (2-6). They would apply a service watermark with a time varying component (2-7), distribute the content (2-8) and periodically generate cryptographic metadata or “manifest” information for that broadcast (2-9).

[0071] A watermark can be used to retrieve the associated, static cryptographic metadata and canonical content. And the time-varying sendee watermarks can be used to retrieve the associated, time-varying cryptographic metadata and canonical content (2- 10).

[0072] Validating Content Authenticity using Watermarking

[0073] FIG. 3 and FIG. 4 summarize how watermarks can be integrated into the content validation process. FIG. 3 illustrates a media object validation scenario. FIG. 4 illustrates a media object canonical processing scenario.

[0074] Media validation uses the cryptographic metadata which may be stored in the media object, distributed with the media object and / or retrievable from the cloud. This object is referred to as a ‘manifest’ in the C2PA standard referenced above.

[0075] Media object validation scenarios

[0076] If the content can be validated by a contained or retrieved manifest, then a successful outcome does not require utilizing canonical content.

[0077] If the media contains a manifest (3-1), the manifest corresponds to a registered distributor (3-2), and the manifest validates the media object (3-3), validation is achieved without reference to a watermark. We would view this as a success scenario.

[0078] Otherwise, if the media object does not contain a watermark (3-4), the media remains unvalidated, this is an exception scenario.

[0079] If the media does contain a watermark (3-4) then the manifest is retrieved from the manifest cloud store (3-5). If this retrieval fails, for example if the URI Authority field provided in the watermark does not correspond to a registered broadcaster, the media object is not validated, an exception scenario.

[0080] If the retrieved manifest’s digital signature does not correspond to an approved broadcaster (3-6), then the media object cannot be validated. Another exception scenario.

[0081] Otherwise, if the manifest’s digital signature is trustworthy (3-6) and the manifest validates the content (3-7), validation is achieved by using the watermark. A success scenario.

[0082] Media object canonical representation scenarios

[0083] If a retrieved manifest (3-5) is trustworthy (3-6) but it does not validate the content (3-7), then it is the view of this paper that the only success scenarios involve canonical processing.

[0084] The decision to perform canonical processing (4-8) can be made by the user posting the content or by the platform supporting the validation logic, depending on the policy being adhered to by the social media platform.

[0085] If the decision is to perform canonical process (4-9), the validator retrieves the canonical content (4-10).

[0086] The previously retrieved manifest (3-5) should always validate the canonical content (4-11). If it does not, it is an error and an exception scenario.

[0087] We view validation of the retrieved canonical content as optional because its retrieval location has been established as trusted through validation of the manifest that contains it (3-6) (3-7).

[0088] Media object canonical processing

[0089] The availability of a canonical version of the media object presents the social media platform with additional success scenario opportunities, including one or more of the following:

[0090] a) Posting the uploaded content together with the retrieved asset, or a link to it.

[0091] b) Providing the uploader with a choice between which version of the content should be posted and posting that version with an appropriate label.

[0092] c) Performing an automated comparison of the uploaded and reference asset to determine the nature and amount of difference between the two.

[0093] d) Automatically replacing the uploaded content with the valid asset content.

[0094] e) Forwarding the uploaded content and the retrieved asset to an internal content moderation process.

[0095] The Above Approach Applied to Live Broadcast

[0096] Low Latency Considerations

[0097] Real-time broadcast and live streaming, often referred to as “glass to glass,” is a process where content is captured through a camera lens and transmitted to a viewer’s screen with minimal delay. Although it is a real-time transmission, it always involves some degree of delay or latency, incidental and / or intentional.

[0098] Live scenarios may be categorized by the degree of latency required. This depends on the content’s nature and the desired viewer experience. Real-time, low latency live is essential for live sports and breaking news, where timely viewing is important. Higher latency can be introduced for any number of reasons. For example, content is often recorded, edited, or processed before broadcast.

[0099] Using digital signatures to protect provenance metadata for ‘glass to glass’ real-time live streaming scenarios is technically challenging. The primary difficulty is that performing a digital signing operation on a Content Delivery Network (CDN) edge server is not adequately secure and performing that operation in a Hardware Security Module (HSM) is unlikely to achieve the low latency desired.

[0100] However, any live scenario where the content is captured downstream, edited, and subsequently posted will introduce an inherent latency sufficient to allow the use of an HSM for provenance metadata protection.

[0101] Live Broadcast News Content Posted to a Social Media Platform

[0102] A 30-minute evening news program is broadcast. The live broadcast is captured and recorded on a device downstream of an HDMI port. A 20-second clip of the news broadcast is created as an MP4 file and posted to a social media platform.

[0103] No manifest can be present with the content since only the elementary stream will make its way beyond the HDMI port. The social media platform can examine the posted video for a watermark, but what would be the success scenario?

[0104] The fragmented MP4 broadcast replica

[0105] The following approach provides a reasonable degree of provenance and authenticity assurance for broadcast news content posted to social media platforms.

[0106] The live news program is broadcast with a watermark consisting of a constant service / asset identifier component and a time-varying index code. See FIG. 5, which shows data hash segments, cryptographic metadata, and watermarks in the scenario of a live news broadcast with a watermark consisting of a constant service / asset identifier component and a time-varying index code. This watermark approach is in use today for delivering metadata for interactive television services and can readily support the retrieval of provenance metadata without the need to modify the watermark itself. See the ATSC 3.0 specifications A / 334, A / 335, and A / 336, discussed below.

[0107] The broadcaster or the network / platform operator on behalf of the broadcaster produces a secure transcoding of specified portions of the live linear broadcast into a fragmented MP4 format - an ‘IMP4 Replica’ of the portion of the linear broadcast for which provenance and authenticity is to be established.

[0108] Periodically a C2PA manifest is produced for this fMP4 Replica. In this design the portion of the Replica that each manifest corresponds to is defined as the Data Hash Segment (DHS). The real-time duration of a DHS defines a minimum lag time behind the linear live edge for the availability of DHS Replica Manifests.

[0109] The Replica itself consists of a sequence of fragmented MP4 segments or chunks for each track, adequate to cover the length of the Data Hash Segment. Each segment or chunk includes auxiliary ‘c2pa’ boxes (defined in the previously mentioned C2PA specification) which can be used by a C2PA validator to validate any portion of the DHS Replica, as described below.

[0110] Apart from the addition of c2pa-specific ISOBMFF boxes, the Replica format is identical to the format in common use for adaptive bitrate streaming, the Common Media Application Format or CMAF. See: ISO / IEC 23000-19:2020, “Information technology - Multimedia application format (MPEG- A) - Part 19: Common media application format (CMAF) for segmented media”.

[0111] The fragmented MP4 replica C2PA manifest

[0112] The fMP4 Replica Manifest is constructed in the exact same way as a C2PA Manifest for audio-video streaming. See section 9.2.3 of the C2PA specification.

[0113] Before the manifest is generated, a DHS initialization segment is produced for the content stored in the DHS Replica. The cryptographic metadata stored in this initialization segment is identical to that specified by C2PA for adaptive bitrate delivery. See Section 9.2.3 of the C2PA specification.

[0114] The c2pa-specific box in each track’s initialization segment will contain the C2PA manifest, which, as is the case for adaptive delivery, must be identical across tracks. The Manifest’s c2pa.bmff.hash assertion will contain CBOR with an array of Merkle rows, one per track.

[0115] In the C2PA specification for adaptive delivery provenance validation, the Merkle tree associated with the entire video stream enables piecewise validation of individual fragment components of the stream without access to the entire stream. The same mechanism enables piecewise validation of arbitrary portions of the DHS using a single DHS manifest.

[0116] Producing a canonical live recording

[0117] If the watermark in the live recording is time-varying, it can be used create a canonical live recording, as shown in FIG. 6.

[0118] The time-varying watermark is used to derive the manifest recovery URL. OTT BINX and OTT EINX are the time indices (Interval Code) corresponding to the start and end of the posted video, respectively.

[0119] A recovery request (6-1) is sent. The DHS Manifest is provided in a recovery response (6-2).

[0120] This recovery request response is identical to the method used today for interactive television. The only change is in the payload of the response from the provenance-authenticity server.

[0121] The DHS corresponding to the manifest is accessed from the Asset Reference Assertion in the DHS Manifest (6-3).

[0122] The fMP4 Replica is used as an authenticated mezzanine format, to produce a canonical MP4 representation of the posted live content. Any portion of the Data Hash Segment can be validated with the DHS Manifest.

[0123] Beginning with the OTT BINX (6-4), the algorithm walks the Data Hash Segments provided in the Recovery Response (6-7) until the EIDX of the posted live recording is reached (6-5).

[0124] The Present Approach Applied to Web Published Content

[0125] Differences without a Distinction

[0126] During validation of content posted to a social media platform, even the slightest alteration to the content can cause it to be flagged as inauthentic. There are multiple scenarios where the content may have been innocently modified by a user, making their edits from a provenance perspective a ‘difference without a distinction’.

[0127] As discussed, we believe the successful outcome for a validation failure to be for the social media platform to recover the original content and use it in one of the ways we outlined. There are cases, however, where this too will result in an unsuccessful outcome.

[0128] Clipped Web Published News Content Posted to Social Media

[0129] One of the most likely such cases is what we are calling “the clipped news segment” scenario.

[0130] Consider the following example. The broadcaster publishes a 30-minute evening news program to their website. The published video file includes cryptographic metadata, and it is watermarked. A user wishes to share a 20-second clip from that 30- minute program. They download the broadcaster published video file, edit it to produce a 20-second clip, and attempt to post it to a social media platform.

[0131] If the editing tool the user used removed the manifest, the social media platform can recover the metadata using the watermark, as described above. Regardless, the file will be flagged as inauthentic. And recovering the canonical version of the content will result in a 30-minute post.

[0132] Producing the canonical news clip

[0133] If the watermark in the clipped news content is time-varying, it is used to derive the manifest recovery URL, a recovery request (1) is sent where BINX is the time index corresponding to the start of the clip. The retrieved DHS Manifest in a recovery response (2) can be used to produce a canonical version of the news clip, as shown in FIG. 6, following the same steps as producing a canonical live recording.

[0134] Since social media platforms transcode posted video into a multitude of targeted formats, it is likely that they would treat the DHS as a canonical mezzanine format to produce a wide variety of device targeted formats. Using fragmented MP4 as a mezzanine format is commonly done. In addition, the MPEG DASH specification provides support to access segments of presentations at a specified media time through the use of an MPD Anchor, using a query parameter "t=" that a client can append to an MPD URL with either an NPT or UTC time. This could be used to access portions of the fMP4 Replica.

[0135] Content credentials overview

[0136] C2PA metadata for an asset conveys assertions such as asset metadata, actions performed, thumbnails, and cryptographic bindings to the content. These assertionsconvey the provenance of the asset. Assertions are combined with additional information to create a claim. The set of assertions referenced by a claim are collected into a logical construct referred to as the assertion store. The claim is digitally signed, creating the claim signature.

[0137] Assertions, Claims, and Claim Signatures and some additional information are combined to form the C2PA Manifest, as shown in FIG. 7. For some formats, the C2PA Manifest may be embedded in the content. For each manifest there is a single assertion store. However, multiple manifests can be associated with an asset, each one representing a specific series of assertions.

[0138] ATSC Watermarking

[0139] In 2016, the US-based digital television broadcast standards organization ATSC, published standards for use of watermarking technology in connection with their development of the ATSC 3.0 (“NextGen TV”) system. These standards provide open specifications for the use of watermarking technology and associated network protocols to deliver arbitrary timed metadata associated with media content to network-connected clients through distribution paths that include media processing (e.g. transcoding) and metadata removal (e.g. HDMI, analog reconversion).

[0140] ATSC’s primary motivating use case for watermarking is enabling access to NextGen TV interactive (two-way) services for viewers who have purchased compatible TVs but who continue to receive broadcast services from other distribution paths, such as STBs, streaming media players, or ATSC 1.0 transmissions. These standards have been commercially deployed in the United States by multiple broadcasters and television equipment manufacturers.

[0141] The technology has also been found to be suitable for use with other broadcast systems. Since 2020, the HbbTV and DVB have published a series of standards that employ ATSC watermarking to enable interactivity and targeted advertising in theirplatforms. These standards are currently being readied for commercial deployment in Germany.

[0142] Description

[0143] The ATSC watermark system is specified in the publicly available standards ATSC A / 334, A / 335, and A / 336, described below. Its function is to deliver arbitrary timed metadata using audio and / or video watermarks embedded into media essence. It supports methods for conveying metadata directly in watermark messages or indirectly, by reference, via carriage of a time-tagged URL that identifies a network resource containing the metadata. The architecture of the ATSC watermark system can be understood using the OSI abstraction model shown in FIG. 8. Taken from bottom to top, essence is the baseband audio or video signal components. The physical layer consists of a stream of raw binary symbols conveyed as audio and video watermarks in the essence. Audio watermarks are conveyed using autocorrelation modulation in the 2.5k-5kHz band. Video watermarks are conveyed using luma modulation in the top two lines of active video. While watermark insertion and detection are performed on baseband (decoded) essence, the system is compatible with a wide range of media processing algorithms applied to watermarked essence, such as low bit-rate coding, that are typically found in digital media distribution. Both audio and video watermark physical layers also pennit watermark energy to be adapted to the content to preserve perceptual quality. In formal testing conducted by ATSC technical committees, the audio wateimark was demonstrated to be capable of suiviving HE-AACv2 encoding at 32 kbps stereo without performance loss while preserving perceptual transparency. The video watermark was demonstrated to be capable of surviving AVC encoding at 2.5 Mbps 1080p / 30.

[0144] The data link layer differs for audio and video watermarks, with the audio wateimark canying a sequence of data cells of 1.5 seconds duration, each carrying a 50- bit data packet along with a synchronization header and BCH error protection. The videowatermark data link layer conveys a 168-bit data packet in each video frame along with a synchronization header, message framing, and CRC error protection.

[0145] The transport layer also differs for audio and video watermarks. For the audio watermark, the transport layer conveys a single packet format, the VP1 payload, that conveys a “tiny URL” encoded into two fields - a server code that identifies a network server and an interval code that identifies a metadata resource on that server associated with the location in the media content where the watermark is embedded. For the video watermark, the transport layer can convey metadata by reference using the VP1 payload or directly, using a variety of messages associated with known broadcast metadata types such as stream events, presentation timestamps, and content identifiers. Server codes are assigned values for which ATSC maintains registry authority.

[0146] The session layer specifies constraints on the arrangement of watermark messages within assets that enable receivers to perform reliable decisioning regarding the arrangement of watermarked content, including when a particular watermarked asset starts and ends and where on the media timeline a given media sample lies. This is particularly important in contexts the content arriving at the receiver has been composed from multiple different sources, such as a “mash-up” of multiple sources or edited version of content.

[0147] The VP1 payload is relied on in the session layer to provide the context boundary for a media asset. A watermark media asset (which can be either an individual program item or a continuous program stream) carries audio or video watermark segments comprised of contiguous, watermarked 1.5 second content intervals with a constant server code value and incrementing interval code values.

[0148] At the application layer, directly conveyed metadata becomes valid at the location in the content where it is placed. For metadata delivery over a network, a RESTfiil application layer protocol between the receiver and a metadata server is specified wherein receivers retrieve arbitrary timed metadata using VP1 payload data.This protocol was adapted by ATSC from an existing 3GPP MBMS protocol. In it, receivers request a metadata resource using a URL constructed from the first VP1 payload that they encounter in a watermark segment. The authority portion of the URL is an Internet hostname determined using DNS resolution of the server code within a second level domain specified by the standard. The path portion of the URL includes the interval code within a predefined template. The response is a multipart / related MIME object containing some protocol-specific metadata objects and some number of additional metadata objects. The protocol-specific metadata includes a mapping of the VP1 interval code value onto a media timeline, boundaries on the media timeline for which each of the additional metadata objects is valid, and guidance to receivers on where and when updates to the metadata objects should be requested. Receivers request subsequent metadata updates only as necessary, in correspondence with the validity periods of metadata objects that they have received and the time periods of watermarked segments of the media timeline of content that they process.

[0149] Any lANA-registered media type can be provided as an additional metadata object. Receivers are expected to route these objects to application-specific handlers and ignore media types that they do not support.

[0150] Suitability

[0151] The ATSC watermark system provides a number of technical capabilities important to the provenance authentication use case.

[0152] Open Architecture

[0153] The ATSC watermark system shares the same underlying architecture as the modem Internet. The system stack is based on publicly available specifications that enable independent development of interoperable implementations of all components. It uses federated DNS namespace management governed by ATSC, a not for-profit, internationally recognized standards development organization. And it does not rely onany siloed or proprietary services, freeing broadcasters to host metadata services on servers of their choosing with the ability to transition to new hosts at will. A typical use case illustrating an open metadata retrieval architecture is provided in FIG. 9.

[0154] Provenance Recovery Client Specification

[0155] This following disclosure describes embodiments of a provenance recovery client and a functional module of a detector toolkit used in provenance authentication workflows. An architectural diagram of a representative provenance authentication workflow is shown in FIG. 10. In the embodiment shown in FIG. 10 the detector toolkit is an Aspect® detector toolkit, the provenance recovery client is an Aspect® provenance recovery client, and the watermark detector is an Aspect® watermark detector. Aspect is a trademark of Verance Coiporation and the Aspect platform is designed to implement the ATSC watermark standards. The general techniques described may also be applied to other watermark technologies, such as those described along with ATSC watermarking in the C2PA specification at: https: / / github.com / c2pa-org / softbinding-algorithm-list which is incorporated herein by reference. Thus, other detector toolkits, provenance recovery clients and watermark detectors may be employed. In this diagram, provenance authentication means a process that analyzes audiovisual media of unknown origin and seeks to discover and authenticate related provenance claims using C2PA metadata and watermarking. Thus, in some embodiments of the disclosure the watermarking technology used is based on ATSC watermarking standards, although other watermarking technologies may be employed. The embodiments of the present disclosure may support a variety of different use cases, including content labeling, content moderation, automated content analysis, or content substitution. These use cases are described in more detail in the inventors’ IBC2024 paper entitled “Interoperable Provenance Authentication of Broadcast Media Using Open Standards-based Metadata, Watermarking and Cryptography” available at https: / / arxiv.org / abs / 2405.12336, the entire contents of which are incorporated by reference herein.

[0156] In the present disclosure, the term media refers to content whose provenance is unknown, assets refers to content whose provenance has been authenticated, and trust signals means authentication information about media and assets.

[0157] The output of the provenance authentication process may be a combination of media with no discoverable provenance, media with provenance claims that could not be authenticated, and assets. The present disclosure is concerned primarily with audio, video, and audiovisual content and with media that may have been subject to alterations and edits of various kinds, so this combination of media and assets may be made up of various components, various time intervals, etc. Establishing the composition of the media and accurately describing each segment and any related provenance information is a key goal of a provenance authentication process.

[0158] The provenance authentication process may be applied to file-based inputs or continuous media streams but, in any case, we assume it is performed on an PC / mobile device application, a PC / mobile device operating system, or a cloud server that does not have significant memory, CPU, or latency constraints.

[0159] The present disclosure may make reference to:

[0160] • C2PA Specification v 1.4 and v2.1 See: https: / / spec.c2pa.Org / specifications / specifications / 2.l / specs / C2PA_Specifica tion.html, which is incorporated herein in its entirety by reference.

[0161] • ATSC A / 334:2024. See: https: / / www.atsc.org / wp- content / uploads / 2024 / 04 / A334-2024-04-Audio-Watermark-Emission.pdf, which is incorporated herein by reference.

[0162] • ATSC A / 335:2024. See https: / / www.atsc.org / wp- content / uploads / 2024 / 04 / A335-2024-04-Video-Watermark-Emission.pdf, which is incorporated herein by reference.

[0163] • ATSC A / 336:2024. See https: / / www.atsc.org / wp- content / uploads / 2023 / 06 / A336-2023-03a-Content-Recovery-in- Redistribution-Scenarios.pdf, which is incorporated herein by reference.

[0164] The disclosed embodiments include implementations that conform to the requirements of the above specifications. Where specifications listed above conflict, some implementations may conform with either specification. Where the present document conflicts with any of the above specifications, implementations will preferably conform to the present document. Terms used in the present document and not defined herein that are defined in any of the above specifications have the meaning given in those specifications.

[0165] Definitions

[0166] ATSC watermark: Means a VP1 Watermark Segment as specified in ATSC A / 336.

[0167] ATSC soft binding assertion: Means a soft binding assertion associated with an ATSC watermark. See: Sections 9.3 and 18.9 of the C2PA specification described above, which is incorporated herein by reference.

[0168] ATSC soft binding watermark: Means an ATSC watermark used for the purpose of providing a C2PA soft binding.

[0169] candidate content: Means content input to a provenance recovery process for authentication.

[0170] provenance recovery client'. Means a client that employs ATSC soft binding watermarks together with a C2PA validator for the purpose of recovering provenance authentication capabilities for C2PA assets.

[0171] soft binding: Has the meaning given in the C2PA Specification.

[0172] soft binding assertion: Has the meaning given in the C2PA Specification.

[0173] soft binding watermark'. Has the meaning given in the C2PA Specification.

[0174] Watermark Media Timeline: Means the Recovery Media Timeline as specified with ATSC A / 336.

[0175] With reference to Fig. 10: OS Services supplies access to typical platform resources such as dynamic memory management, persistent storage, and network servers;

[0176] The C2PA Validator has the function described in the C2PA Specification. It is assumed to be configured with Trost Lists selected by the implementor or user of the provenance authentication process. The provenance authentication function uses it to locate manifests in the media, attempt to validate those manifests, and use hard bindings in valid manifests to authenticate input media;

[0177] Media Decoders convert media from media transport and encoding formats into demultiplexed and decoded (i.e., baseband) media components suitable for watermark detection;

[0178] Aspect Watermark Detectors are the AWD-SID and VWD-SID functions that detect VP1 Watermark Segments in baseband audio and video components;

[0179] The function performed by the Provenance Recovery Client (or PRC) is the primary subject of this document and is described in detail in the following sections. In relation to other elements of FIG. 10, it uses the Aspect Watermark Detector to detect VP1 Watermark Segments in media, the OS Services to retrieve metadata and manifests associated with them from network servers and store them locally, and the C2PA Validator to validate retrieved manifests. It establishes the presence and details of valid ATSC soft-binding assertions in media and makes this information available for use in the provenance authentication process.

[0180] Provenance Recovery Client

[0181] • Interfaces

[0182] i. PRC to Provenance Authentication

[0183] The PRC interface to the provenance authentication process provides functions for configuration and reporting of soft-binding assertions.

[0184] Configuration

[0185] The configuration interface includes establishing connections between the PRC and other system functions (e.g., watermark detectors, provenance authentication process) which may be executing in separate processes. Watermark detector instances are referenced by unique identifiers associated with the media component instances that they process.

[0186] The configuration interface also provides methods for the provenance authentication process to notify the PRC that end-of-media has been reached. It also includes configuration of the PRC policy for real-time (true / false). This policy indicates whether the media and provenance authentication output is being presented to a viewer in real-time, in which case the provenance recoveiy client should minimize the latency with which results are reported. In non-real-time operation, provenance recovery records may be delivered with substantial latency, including after end-of-media is indicated by the provenance authentication process.

[0187] The PRC configuration is not expected to change during the lifecycle of a PRC instance. In some implementation, certain PRC configuration settings may be fixed at build time rather at runtime.

[0188] Reporting

[0189] The reporting interface provides means for the provenance authentication process to obtain lists of provenance recovery records, watermark segment records, and manifests from the PRC.

[0190] A provenance recovery record is a data structure described in Table 1. Detailed descriptions of how these fields are populated are provided in the Functional Description of the PRC section of this document.Table 1. Provenance recovery record data structure.

[0191] The Timestamp data type must be capable of representing media times on RFC 2326 (See: https: / / www.rfc-editor.org / rfc / rfc2236) “normal play time” (NPT) and ISO 8601 (See: https: / / www.iso.org / iso-8601-date-and-time-fomiat.html) timelines. It may also be useful for it to be capable of representing “open ended” intervals (e.g., where end time is absent, “infinity” or “NaN”) in real-time use cases where a record is provided for a segment that is ongoing; e.g., where the end time has not been reached at the live edge. (Alternatively, the end timestamps could record the latest confirmed time for the segment and a separate field could be included in the record indicating whether the segment is open / ongoing or closed / ended.)

[0192] The Component Description data type is a data structure described in Table 2.Detailed descriptions of how these fields are populated are provided in the Functional Description of the PRC section of the present disclosure. Table 2. Component Description data structure.

[0193] The data structure for Watermark Segment Records are described in Table 4.Manifests are strings in the JSON format used by c2patool.

[0194] ii. PRC to Watermark Detector

[0195] Watermark detector events provide the PRC with the VP1 watermark segment boundaries and the values and positions of VP1 payloads in those segments. Time values are given on the media timeline associated with the input media. The interface and function of Aspect watermark detectors are specified in their respective documents.

[0196] iii. PRC to C2PA Validator

[0197] A C2PA Validator is expected to be provided and configured (e.g., with a trust list) by the provenance authentication process. It is expected to provide functions for processing decoupled manifests provided by the PRC, including manifest validation in accordance with C2PA specifications and manifest parsing into the JSON format compliant with the schema used by c2patool (See: https: / / opensource.contentauthenticity.org / docs / manifest / manifest-examples / .)

[0198] iv. PRC to OS Services

[0199] A platform-independent API to typical OS services such as IP networking, dynamic memory management, and persistent storage. (The existing ART OS ServicesAPI is expected to be sufficient.)

[0200] • Function (Non-realtime Mode)

[0201] v. State DataTable 3. State data of the PRC function (Non-realtime Mode).Table 4. Watermark segment record data structure.

[0202] vi. Algorithm

[0203] The PRC non-realtime algorithm has three main steps that are performed sequentially. These are: (1) Build Watermark Segment List; (2) Determine Watermark Segment Validity; (3) Build Provenance Record List.

[0204] Build Watermark Segment List

[0205] In embodiments using the ATSC watermark, this step implements ATSC A / 334, A / 335, and A / 336. The input is watermark detector events and the output is a Watermark Segment List where all fields of the Watermark Segment Records that can be determined from the recovery response have been populated.

[0206] 1. Process all components through watermark detectors until end-of-media is reached.

[0207] 2. Add one Watermark Segment Record to the Watermark Segment List for each WSS / WSE pair reported by a watermark detector, populating the following fields:

[0208] a. Watermark Detector (detector that generated the WSS / WSE pair)

[0209] b. Server Code (from WSS)

[0210] c. Interval Code Start (from WSS)

[0211] d. Interval Code Start Fraction (from WSS)

[0212] e. Interval Code End (from WSE)

[0213] f. Interval Code End Fraction (from WSE)

[0214] g. Validity (initialize to Valid)

[0215] h. Pointers should be initialized to null

[0216] i. Strings should be initialized to empty

[0217] 3. Retrieve recovery responses for each Watermark Segment Record on theWatermark Segment List.

[0218] a. Start with the first SC / IC of the segment. If the recovery response derived from a segment has validity period that does not span thatsegment, follow the nextURL until the segment is spanned. Split the Watermark Segment Record into multiple Watermark Segment Records as needed to achieve one-to-one relationship between segments and C2PA Manifest Retrieval URLs (entire URL must be identical).

[0219] b. As an optional method to reduce network requests, recovery responses may be stored in a local cache. Before issuing a recovery request, check the cache for a recovery response that is valid for the segment. Only the thisComponent description should be used for validity matching (not otherComponents) to avoid attacks using spoofed otherComponent descriptions.

[0220] 4. Populate the following fields of each Watermark Segment Record with data from the recovery responses (this can be done as recovery responses are retrieved):

[0221] a. WMT Start (from WSS and anchor)

[0222] b. WMT End (from WSE and anchor)

[0223] c. C2PA Manifest Retrieval URL (from recovery response)

[0224] d. Recovery Response (if cache is implemented)

[0225] e. Validity (set to NoManifestURL if recovery response does not include a C2PA Manifest Retrieval URL and to ManifestRecoveiyFailure if there is C2PA Manifest Retrieval URL but it could not be retrieved)

[0226] Determine Watermark Segment Validity

[0227] This step establishes the validity state of each of the Watermark Segment Records. It implements all mandatory parts of the Provenance Recovery section of the ATSC Soft Binding Assertion Interoperability Specification.

[0228] 5. Retrieve the C2PA manifest store for all unique C2PA Manifest RetrievalURLs in a Watermark Segment Record and store them in the Manifest List.

[0229] 6. Validate each manifest store in the Manifest List and store the results in theManifest Validation Result List.

[0230] 7. For each Watermark Segment Record with a Manifest Retrieval URL that has Validity state equal to Valid:

[0231] a. Populate the Manifest and Manifest Validation Results fields of theWatermark Segment Record with a reference to the associated items in the Manifest and Manifest validation Results Lists. (Note that multiple Watermark Segment Records can be associated with the same manifest store.)

[0232] b. Set Validity state to BadManifest if the Manifest Validation Results of the Manifest associated with this Watermark Segment Record indicate that the Manifest is not valid and end step 7 processing of this Watermark Record Segment

[0233] c. If the C2PA Manifest Retrieval URL specifies an ATSC Soft BindingAssertion Scope URN, locate the JUMBF box in the Manifest whose C2PA URN is equal to the ATSC Soft Binding Assertion Scope URN. If no such JUMBF box exists, set the Watermark Record Segment Validity state to BadSoftBindingAssertionScope and end step 7 processing of this Watermark Record Segment.

[0234] d. If the C2PA Manifest Retrieval URL specifies an ATSC Soft BindingAssertion Scope URN, locate the outermost JUMBF box of the Manifest.

[0235] e. Locate the ATSC Soft Binding Assertion nearest the end of thisJUMBF box containing a Recovery File that includes a thisComponent or otherComponent whose server code matches that of this Watermark Segment Record and whose time-map corresponds to an interval code range that overlaps with the interval code range of the Watermark Segment Record. For purposes of determining the interval code range that corresponds to the time-map, the provenance recovery client should employ data from the Recovery File in the ATSC Soft Binding Assertion, not the Recovery File in the recovery response.

[0236] i. If no such Recovery File exists, set the Watermark RecordSegment Validity state to NoMatchingSoftBindingAssertion and end step 7 processing of this Watermark Record Segment.

[0237] ii. If the overlap is partial, then divide the Watermark SegmentRecord into multiple Watermark Segment Records as necessary to separate overlapping and non-overlapping ranges. (If the overlap is within the range of watermark detector boundary error, it may be acceptable to ignore or discard small measurement errors. A tolerance value should be included in the implementation and tuned experimentally.).

[0238] 1. New Watermark Segment Records created by division can preserve all data elements fromparent except the start and end elements, which must be recalculated to include only the nonoverlapping intervals. The start and end elements for the overlapped Watermark Segment Record must also be recalculated to include only the overlapped interval.

[0239] 2. Queue newly created, non-overlapping segments for processing by step 7. Continue processing overlapped segment through the following steps.

[0240] f. Set the Asset Start Time and Asset End Time in accordance with theAsset Media Timeline using located component information from this Recovery File.

[0241] g. Set the Soft Binding Assertion URN to the C2PA URN of this ATSCSoft Binding Assertion.

[0242] Build Provenance Record List

[0243] A Provenance Recovery record provides a description of a “clip” of one or more time-aligned components taken from an asset. It is more useful to the provenance authentication process than a collection of Watermark Segment Records, which may come from different assets and may be misaligned in time. This step converts valid Watermark Segment Records into Provenance Recoveiy records via a sequence of merges and divisions.

[0244] 8. For each Watermark Segment Record that has Validity state equal to Valid:

[0245] a. If Provenance Recovery List contains a Provenance Recovery Record that has matching Manifest and overlapping Asset Start / End Time intervals:

[0246] i. If the overlap is partial, then create multiple ProvenanceRecovery Records as necessary to separate overlapping and non-overlapping ranges. (If the overlap is within the range of watermark detector boundary error, it may be acceptable to ignore or discard small measurement errors. A tolerance value should be included in the implementation and tuned experimentally.).

[0247] 1. New Provenance Recoveiy Records created by division can preserve all data elements from parent except the start and end elements, which must be recalculated to include only the nonoverlapping intervals. The start and end elements for the overlapped Provenance Recovery Record and its components must also be recalculated to include only the overlapped interval.

[0248] b. If Provenance Recoveiy List does not contain a Provenance RecoveryRecord that has matching Manifest and overlapping Asset Start / End Time intervals, create a new Provenance Recoveiy from this component.

[0249] c. Populate and revise fields of newly created and modified ProvenanceRecoveiy Records to reflect details of this Watermark Segment Record as a component as follows:

[0250] i. Set Media Start to the earlier of the two Media Start times.

[0251] ii. Set Media End to the later of the two Media End times.

[0252] iii. Set Asset Start to the earlier of the two Asset Start times.

[0253] iv. Set Asset End to the later of the two Asset End times.

[0254] v. Add a new Component Description to the Component List with fields set to corresponding values from the Watermark Segment Record.

[0255] vi. Set the following Provenance Recovery Record to the value from the Watermark Segment Record:

[0256] 1. Manifest Retrieval URI

[0257] 2. Manifest

[0258] 3. Soft Binding Assertion URN

[0259] 4. Manifest Validation Result

[0260] 5. Set the Asset Reference Assertion URI to the lastAsset Reference Assertion in the Manifest of the Watermark Segment Record.

[0261] 9. Output Provenance Recovery List, Watermark Segment List, and ManifestList to the provenance authentication process.

[0262] It is understood that the various embodiments of the present invention may be implemented individually, or collectively, in devices comprised of various hardware and / or software modules and components. These devices, for example, may comprise a processor, a memory unit, an interface that are communicatively comiected to each other, and may range from desktop and / or laptop computers, to consumer electronic devices such as media players, mobile devices, and the like. For example, FIG. 11 illustrates a block diagram of a device 1000 within which the various disclosed embodiments may be implemented. The device 1000 comprises at least one processor 1002 and / or controller, at least one memory 1004 unit that is in communication with the processor 1002, and atleast one communication unit 1006 that enables the exchange of data and information, directly or indirectly, through the communication link 1008 with other entities, devices and networks. The communication unit 1006 may provide wired and / or wireless communication capabilities in accordance with one or more communication protocols, and therefore it may comprise the proper transmi tter / recei ver antennas, circuitry and ports, as well as the encoding / decoding capabilities that may be necessary for proper transmission and / or reception of data and other information.

[0263] Referring back to FIG. 11 the device 1000 and the like may be implemented in software, hardware, firmware, or combinations thereof. Similarly, the various components or sub-components within each module may be implemented in software, hardware, or firmware. The connectivity between the modules and / or components within the modules may be provided using any one of the connectivity methods and media that is known in the art, including, but not limited to, communications over the Internet, wired, or wireless networks using the appropriate protocols.

[0264] Various embodiments described herein are described in the general context of methods or processes, which may be implemented in one embodiment by a computer program product, embodied in a computer-readable medium, including computerexecutable instructions, such as program code, executed by computers in networked environments. A computer-readable medium may include removable and non-removable storage devices including, but not limited to, Read Only Memory (ROM), Random Access Memory (RAM), compact discs (CDs), digital versatile discs (DVD), etc. Therefore, the computer-readable media that is described in the present application comprises non- transitory storage media. Generally, program modules may include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of program code for executing steps of the methods disclosed herein. The particular sequence of such executableinstructions or associated data structures represents examples of corresponding acts for implementing the functions described in such steps or processes.

[0265] The foregoing description of embodiments has been presented for purposes of illustration and description. The foregoing description is not intended to be exhaustive or to limit embodiments of the present invention to the precise form disclosed, and modifications and variations are possible in light of the above teachings or may be acquired from practice of various embodiments. The embodiments discussed herein were chosen and described in order to explain the principles and the nature of various embodiments and its practical application to enable one skilled in the art to utilize the present invention in various embodiments and with various modifications as are suited to the particular use contemplated. The features of the embodiments described herein may be combined in all possible combinations of methods, apparatus, modules, systems, and computer program products.

Claims

1. WHAT IS CLAIMED IS:

1. A method for providing provenance information for media comprising: receiving input media having components with soft binding watermarks embedded therein; detecting and processing the watermarks using watermark detectors; building a Watermark Segment List using the output of the watermark detectors; creating a Watermark Segment Record; establishing the validity of each Watermark Segment Record; building a Provenance Recovery Record List, including ProvenanceRecovery Records which each provide a description of a clip of one or more time- aligned component taken from a media asset by converting Watermark Segment Records into Provenance Recovery records; and outputting the Provenance Recovery List and Watermark Segment List, to a provenance authentication process.

2. The method according to claim 1 wherein building a Watermark Segment List comprises; processing all components through watermark detectors until end-of- media is reached; adding one Watermark Segment Record to the Watermark Segment List for each Watermark Segment Start (WSS) / Watermark Segment End (WSE) pair reported by a watermark detector; retrieving recovery responses for each Watermark Segment Record on the Watermark Segment List; populating a plurality of fields of each Watermark Segment Record with data from the recovery responses;3. The method according to claim 2 wherein establishing the validity of each Watermark Segment Record comprises: retrieving a C2PA manifest store for all unique C2PA Manifest Retrieval URLs in a Watermark Segment Record and storing them in a Manifest List; validating each manifest store in the Manifest List and store the results in the Manifest Validation Result List; and for each Watermark Segment Record with a Manifest Retrieval URL that has Validity state equal to Valid populating the Manifest and Manifest Validation Results fields of the Watermark Segment Record with a reference to the associated items in the Manifest and Manifest validation Results Lists.

4. The method according to claim 3 wherein the building a Provenance Recovery Record List comprises: for each Watermark Segment Record that has Validity state equal to Valid: a. If Provenance Recovery List contains a Provenance Recovery Record that has matching Manifest and overlapping Asset Start / End Time intervals: i. If the overlap is partial, then create multipleProvenance Recovery Records as necessary to separate overlapping and non-overlapping ranges. (If the overlap is within the range of watermark detector boundary error, it may be acceptable to ignore or discard small measurement errors. A tolerance value should be included in the implementation and tuned experimentally.).

1. New Provenance Recovery Records created by division can preserve all data elements from parent except the start and end elements, which must be recalculated to include only the non-overlapping intervals.The start and end elements for the overlapped Provenance Recovery Record and its components must also be recalculated to include only the overlapped interval. b. If Provenance Recovery List does not contain a Provenance Recovery Record that has matching Manifest and overlapping Asset Start / End Time intervals, create a new Provenance Recovery from this component. c. Populate and revise fields of newly created and modified Provenance Recovery Records to reflect details of this Watermark Segment Record as a component as follows: i. Set Media Start to the earlier of the two Media Start times. ii. Set Media End to the later of the two Media End times. iii. Set Asset Start to the earlier of the two Asset Start times. iv. Set Asset End to the later of the two Asset End times. v. Add a new Component Description to the Component List with fields set to corresponding values from the Watermark Segment Record. vi. Set the following Provenance Recovery Record to the value from the Watermark Segment Record:

1. Manifest Retrieval URI2. Manifest3. Soft Binding Assertion URN4. Manifest Validation Result5. Set the Asset Reference Assertion UR Set the Asset Reference Assertion URI as follows:a. If the C2PA Manifest Retrieval URL of the Watermark Segment Record specifies an ATSC Soft Binding Assertion Scope URN and an Asset Reference Assertion is present in the JUMBF box of the Manifest of the Watermark Segment Record whose C2PA URN is equal to the ATSC Soft Binding Assertion Scope URN, then set the Asset Reference Assertion URI to that Asset Reference Assertion’s “uri” field value. b. Otherwise, set the Asset Reference Assertion URI to “uri” field value of the last Asset Reference Assertion in the Manifest of the Watermark Segment Record, with the constraint that the selected Asset Reference Assertion must be contained in either the same manifest as or an ancestor manifest (in the “parentOf ’ chain of ingredient manifests) of the manifest referenced by the Soft Binding Assertion URN of the Watermark Segment Record.

5. The method according to claim 1 wherein the Soft binding watermarks are ATSC soft binding watermarks.

6. A system for providing provenance authentication of media content comprising: provenance authentication unit for receiving and analyzing media content to discover and authenticate related provenance claims using C2PA metadata and generating an output including, when possible, authenticated content and tiust signals;operating system services unit providing access to platform resources including dynamic memory management, persistent storage, and network servers;C2PA validator used by the provenance authentication unit to locate manifests in the media, validate those manifests, and use hard bindings in valid manifests to authenticate input media; media decoder for converting media from media transport and encoding formats into demultiplexed and decoded media components; watermark detector for detecting watermarks in audio or video components of the media; and recovery client for using the watermark detector to detecting watermark segments in the media, using the operating system services unit to retrieve metadata and manifests associated with the watermark segments from network servers and store them locally, and using the C2PA validator to validate retrieved manifests, wherein the details of valid soft- binding assertions are established and made available to the provenance authentication unit.

7. The system according to claim 6 wherein, based on the analyses, the provenance authentication unit output includes one or more of: media with no discoverable provenance, media with provenance claims that could not be authenticated, and authenticated content.

8. The system according to claim 6 wherein the provenance authentication unit establishes the composition of the media, and describes segments of the media and related provenance information.

9. The system according to claim 6 wherein received content has been subjected to alterations.

10. The system according to claim 6 wherein the watermark detector is an ATSC watermark detector that detects VP1 watermark segments in audio and video components.

11. The system according to claim 6 wherein the recovery client further builds a watermark segment list, determines watermark segment validity, and builds a provenance record list.

12. The system according to claim 6 wherein the recoveiy client builds a watermark segment list by: processing all components through watermark detectors until end-of-media is reached; adding one Watermark Segment Record to the Watermark Segment List for each Watermark Segment Start (WSS) / Watermark Segment End (WSE) pair reported by a watermark detector; retrieving recovery responses for each Watermark Segment Record on theWatermark Segment List; and populating a plurality of fields of each Watermark Segment Record with data from the recovery responses.

Images