Method, device, and system for preparing content of a media stream

Abstract

The disclosure notably relates to methods and systems for preparing content of a media stream. The methods and systems allow to reduce the amount of secondary content that would otherwise be displayed, in the media stream, without the selection process of the above-referred disclosure.

Description

[0001] HE Ref.: 269353

[0002] Method, device, and system for preparing content of a media stream

[0003] [Field of technology]

[0004] The invention relates to the field of preparing content of a media stream and to inserting content into a media stream. The invention may be used to, though not limited to, assist in inserting content into a media stream in a manner tailored to the initial content of a media stream.

[0005] [Background]

[0006] Various techniques exist for preparing contents of a media stream. These techniques often involve inserting a specific kind of content at specific temporal references, such as time spans or time points, of a media stream. Various techniques also exist for determining the kind of content to be inserted.

[0007] In view of the above, there is a constant need for improved solutions to assist in determining the kind of contents to be inserted and inserting these contents into a media stream.

[0008] [Summary]

[0009] The present invention aims at addressing, at least partially, the above-mentioned need. The invention includes a method, a device, and a system, as defined in the independent claims. Further embodiments are defined in the dependent claims.

[0010] In one embodiment, a method is provided for preparing content of a media stream. The method comprises: sending, by a first processing system, hereinafter referred to as "content management system", to a second processing system, hereinafter referred to as "secondary content exchange system", information, hereinafter referred to as "secondary content opportunity information", wherein the secondary content opportunity information indicates an association between content, hereinafter referred to as "primary content", appearing in the media stream and a temporal reference at which the primary content appears in the media stream; receiving, by the content management system from the secondary content exchange system, in response to sending the secondary content opportunity information, information, hereinafter referred to as "slot information", indicating an interest in inserting content, hereinafter referred to as "secondary content", into the media stream; and inserting into the media stream, by the content management system, and in response to receiving the slot information: information, hereinafter referred to as "announcement information", based on the received slot information and to be displayed at a client device, and an indicator, hereinafter referred to as "interactivity indicator", to be displayed at the client device and indicating to a user of the client device that the user, through the client device, can transmit feedback in relation to the announcement information. HE Ref.: 269353

[0011] The method in the above-referred embodiment generally allows for preventing the displaying of secondary content that would otherwise be displayed without the above selection mechanism. This reduces the burden on the network and the burden on the client device. Thereby, processing resources and energy can be saved. Saving energy is particularly beneficial for battery-powered devices.

[0012] Further advantages and details will become apparent from the following detailed description of some embodiments of the invention.

[0013] [Brief description of the drawings]

[0014] Figure 1 shows a flowchart of a method for preparing content of a media stream according to a first embodiment of the invention.

[0015] Figure 2 shows an example of a second embodiment of the invention.

[0016] Figure 3 shows a diagram depicting an example of a third embodiment of the invention.

[0017] Figure 4 shows a diagram depicting an example of a fourth embodiment of the invention.

[0018] Figure 5 shows a diagram depicting an example of a fifth embodiment of the invention.

[0019] Figure 6 shows a diagram depicting aspects of an eighth embodiment of the invention.

[0020] Figure 7 shows a diagram depicting an example of a ninth embodiment of the invention.

[0021] Figure 8 shows a diagram depicting an example of a tenth embodiment of the invention.

[0022] Figure 9 shows a diagram depicting an example of an eleventh embodiment of the invention.

[0023] Figure 10 shows a diagram depicting an example of a twelfth embodiment of the invention.

[0024] Figure 11 shows a diagram depicting an example of a thirteenth embodiment of the invention.

[0025] Figure 12a shows a diagram depicting an example of a fourteenth embodiment of the invention.

[0026] Figure 12b shows a diagram depicting an example of a fifteenth embodiment of the invention. Figure 13a shows a diagram depicting an example of a sixteenth embodiment of the invention. Figure 13b shows a diagram depicting an example of a seventeenth embodiment of the invention.

[0027] [Detailed description]

[0028] The present invention shall now be described in conjunction with specific embodiments. The specific embodiments serve to provide the skilled person with a better understanding but are not intended to in any way restrict the scope of the invention, which is defined by the appended claims. In particular, the embodiments described independently throughout the description can be combined to form further embodiments to the extent that they are not mutually exclusive.

[0029] Figure 1 shows a flowchart of a method for preparing content of a media stream according to a first embodiment of the invention. The method comprises: sending 103, by a first processing system 101, here referred to as "content management system", to a second processing system 102, here referred to as "secondary content exchange system", information, here referred to as "secondary content opportunity information", wherein the secondary content opportunity information indicates an HE Ref.: 269353 association between content, here referred to as "primary content", appearing in the media stream and a temporal reference at which the primary content appears in the media stream; receiving 104, by content management system 101 from secondary content exchange system 102, in response to sending 103 the secondary content opportunity information, information, here referred to as "slot information", indicating an interest in inserting content, here referred to as "secondary content", into the media stream; and inserting 105 into the media stream, by content management system 101, and in response to receiving 104 the slot information: information, here referred to as "announcement information", based on the received slot information and to be displayed at a client device, and an indicator, here referred to as "interactivity indicator", to be displayed at the client device, the interactivity indicator indicating to a user of the client device that the user, through the client device, can transmit feedback in relation to the announcement information.

[0030] Sending information may mean that a message comprising the information is sent by content management system 101 to secondary content exchange system 102. Stated differently, secondary content exchange system 102 acquires, receives, or obtains a message comprising the information from content management system 101. Sending the message may imply transmitting, transferring, transporting, or delivering the message. Content management system 101 and secondary content exchange system 102 may be part of a metadata information delivery network (MDN). The term MDN may refer to a system of distributed servers configured to interact with each other, thereby transmitting data to a client device, receiving data from the client, and processing data received from the client device. Moreover, the messages may be transmitted and received using protocols, e.g., the hypertext transfer protocol secure (HTTPS). For instance, content management system 101 may send information to secondary content exchange system 102 using the HTTPS 'Post' method or the HTTPS 'PUT' method. The message sent by the transmitting system may be transmitted over the internet using, for example, the TCP / IP protocol.

[0031] Information, for example, secondary content opportunity information, may refer to a result of processing data. The processing may be done by a computer. Information may also refer to data that has meaning. Alternatively, information may refer to a plurality of bits that represents data that has meaning. If the bits are processed by a computer, the computer may be able to present to a user via a graphical user interface (GUI) e.g., a display of the computer, a graphical representation of the information. An example of secondary content opportunity information is discussed below with reference to Figure 2.

[0032] Moreover, based on processing the secondary content opportunity information, secondary content exchange system 102 may be able to generate another type of information, e.g., slot information, as discussed further below. HE Ref.: 269 353

[0033] An example of information is data, for example, a string of bits, comprised in the body of an HTTPS request message used in the HTTPS POST method. Another example of information is data comprised in the body of an HTTPS request message used in the HTTPS PUT method. Moreover, the secondary content opportunity information may be meta data. Moreover, the secondary content opportunity information may be an index generated by video indexing, wherein further non-limiting details about video indexing are also discussed further below. A specific example and further details about the secondary content opportunity information are detailed throughout the below embodiments.

[0034] Content management system 101 may comprise one device or a plurality of devices that are able to perform computations, process data, or process information, thereby generating further information or data. A plurality of interconnected devices may also be referred to as a distributed system. An example of a device being able to perform computations and data processing is a server or computer. The server or the computer may be connected to the internet and configured to generate a message comprising the secondary content opportunity information, wherein the message may be sent to secondary content exchange system 102. An example of a content management system 101 is a server or a computer. Content management system 101 may be implemented as a software entity virtualized in a cloud system.

[0035] Secondary content exchange system 102 may comprise another piece of hardware architecture, e.g., a distributed system comprising a plurality of interconnected devices (e.g., servers), or a stand-alone processing device, for example, a server, that may be configured to receive and transmit messages. Messages may be received via a wireless or wired connection, or a combination thereof. Messages may be received from another device, for example, via the internet or telecommunications networks. An example of secondary content exchange system 102 is a server or a computer. Secondary content exchange system 102 may be implemented as software entity virtualized in a cloud system.

[0036] As mentioned above, content management system 101 and secondary content exchange system 102 may be part of a MDN. As also mentioned above, a MDN may be a system of distributed servers configured to interact with each other, thereby transmitting data to a client device, receiving data from the client, and processing data received from the client device. Content management system 101 may be a first server connected to, or being part of, the MDN. Secondary content exchange system 102 may be a second server connected to, or being part of, the MDN. Alternatively, content management system 101 or secondary content exchange system 102 may be connected to an MDN.

[0037] As discussed above, the secondary content opportunity information indicates an association between content, also referred to as "primary content", appearing in the media stream, and a temporal reference at which the primary content appears in the media stream. HE Ref.: 269 353

[0038] Indicating an association may mean showing, pointing out, or signaling an association. The association may be indicated to secondary content exchange system 102. The secondary content opportunity information may indicate the association by means of data which, if the data is processed by secondary content exchange system 102, results in further data that represents the association. For example, the secondary content opportunity information may be a pointer to an entry in a data set, wherein the data set comprises a plurality of entries and wherein the data set may be linked to (i.e., may correspond to) a temporal reference. As another example, the secondary content opportunity information may indicate the association by comprising data which, if the data is processed by secondary content exchange system 102, results in further data that represents the primary content, e.g., by an index, e.g., by a string of bits that, if converted based on ASCII notation, describes the primary content (such as 'tornado', 'blizzard', 'flooding', 'Venice'). As a specific example, the secondary content opportunity information may be a string of bits, comprised in the body of an HTTPS request in an HTTPS POST or an HTTPS PUSH method, wherein the request may be transmitted from content management system 101 to secondary content exchange system 102. After receiving the HTTPS request, secondary content exchange system 102 may process the string of bits and thereby compute, determine, generate, or calculate first data, wherein the first data may represent the primary content, and second data, wherein the second data may be linked to the first data and may represent a temporal reference.

[0039] An association between content and a temporal reference may mean a relation, a relationship, a linkage, or a connection between content and a temporal reference. In other words, an association between primary content and a temporal reference may comprise data which, if processed by secondary content exchange system 102, represents a one-to-one relationship (1:1), a one-to-many relationship (l:n), or a many-to-many relationship (n:m) of a temporal reference and primary content. For example, a temporal reference may be linked to one primary content. This may be an example of a one-to-one relationship.

[0040] A temporal reference may be data that indicates a time point, a time span, or a plurality of time spans included in the media stream. A time span may include a plurality of time points. A time point may be a time instance at which one video frame out of a plurality of video frames that form the media may be displayed. For instance, if a video is played back at 24 frames per second, and the video runtime is 1 second, a time point may be the time instance where the first video frame of 24 video frames is displayed. For example, time equals 1 / 24 seconds after start of the video. The plurality of video frames may be displayed consecutively. The temporal reference may be the time point, time span, or plurality of time spans. As an example of a time span, a temporal reference may indicate a timespan starting at 00:00, i.e., minute 00 and second 00, to 10:00, i.e., minute 10 and second 00, of a media stream, and the primary content may be 'Lake Michigan'. The primary content being 'flooding' may reflect that the media stream in the above time span contains media data related to 'Lake Michigan'. HE Ref.: 269 353

[0041] A media stream may comprise audio data, video data, or a combination thereof. A media stream may comprise a plurality of parts. The audio data and / or video data may be continuously transmitted from a server to a client device, for example, a smart TV or a computer connected to the internet. The audio data or the video data need not be transmitted but may be transmitted. In other words, the audio or video data may be transmittable. A media stream may be transmitted between a server and a client device using a transmission protocol. Examples of transmission protocols comprise HLS (HTTP Live Streaming), MPEG-DASH (Dynamic Adaptive Streaming over HTTP), RTMP (Real Time Messaging Protocol), WebM, mp4, MPEG-TS and DVB-DASH. The audio or video data may comprise a plurality of parts of the audio or video data. The entirety of the plurality of parts of the audio and / or video data may form the media stream. The data may be transmitted over the internet, or any other type of network or by any other means of wireless or wired transmissions. Examples of a transmission comprise a video on demand transmissions, linear TV transmissions, start-over TV transmissions, or catch-up TV transmissions.

[0042] In other words, primary content may refer to what a viewer of the media stream sees if the data comprised in the media stream is processed by a client device receiving the media stream and the result of the processing is displayed by the client device to a user. In other words, the client device may generate an audiovisual content that can be displayed graphically, e.g., by means of a media player at the client device. The client device may display the audiovisual content. In other words, primary content may refer to data that, if processed, results in audiovisual content that can be displayed by a client device, after receiving one or a plurality of parts of the media stream. For example, a smart TV may receive the audio and / or video data comprised in the media stream as data in mp4 format, process these, thereby computing audiovisual content. An example of audiovisual content is a sequence of consecutive video frames. The video frames may have a certain resolution and frame rate.

[0043] In other words, primary content appearing in a media stream encompasses primary content that would be displayed at a client device based on the media stream based on processing the primary content, for example in the manner discussed above. Displaying primary content based on the media stream may comprise the steps of (i) by the client device, receiving the media stream, (ii) by the client device, generating audiovisual data by processing the media stream, and (iii) by the client device, displaying the audiovisual data, e.g., by means of a media player.

[0044] The above embodiment further comprises, as indicated above, receiving 104, by content management system 101 from secondary content exchange system 102, in response to sending 103 the secondary content opportunity information, slot information indicating an interest in inserting secondary content into the media stream. HE Ref.: 269353

[0045] Slot information may refer to data. The data may be comprised in a message. The message may be transmitted from secondary content exchange system 102 to content management system 101. The transmission of the message may be based on any wireless or wired transmission protocol. Examples of these transmission protocols may be any protocol used for sending information over the internet. For example, secondary content exchange system 102 may use the HTTPS 'Post' or the HTTPS 'Put' method, wherein a HTTPS request message comprises a body and wherein the body comprises data indicating the above interest. Slot information may be returned by secondary content exchange system 102. Slot information may for example contain a picture or a logo or a plurality of pictures or logos. Slot information may, as another example, contain a text description and a slot opportunity instruction. A picture or a logo, or a plurality thereof may be displayed as part of the announcement information, wherein further details about the announcement information is discussed further below.

[0046] Receiving the slot information in response to sending the secondary content opportunity information may comprise the steps of, by secondary content exchange system 102, receiving the secondary content opportunity information, and, if secondary content exchange system 102 has received the content opportunity information, sending, by secondary content exchange system 102, the slot information to content management system 101.

[0047] Receiving the slot information by content management system 101 may mean that content management system 101 receives, from secondary content exchange system 102, a message comprising the slot information. For example, the message may be a HTTPS request message comprising the slot information in the message body as discussed above.

[0048] The secondary content opportunity information indicating an interest in inserting content may mean that the secondary content opportunity information comprises data that indicates or represents the interest in inserting content. The data may be a bit string, a bit, or a data field.

[0049] Slot information indicating an interest in inserting secondary content may mean that the slot information comprises data which, if the data is processed by content management system 101, indicates a secondary content provided by a content provider, wherein the content provider may be interested in having the secondary content inserted into the media stream. The content provider may be interested in having the secondary content inserted at a specific temporal reference.

[0050] The secondary content provided by a content provider may be computed or determined based on the secondary content opportunity information. Thereby, the secondary content may be linked to the association of primary content and temporal reference, or the primary content. For example, the secondary content may be determined based on the primary content, or the association indicated by the secondary content opportunity information. For example, secondary content may be determined HE Ref.: 269 353 based on the primary content by, for example, determining semantic similarity between the primary content, the content provider and the secondary content based on applying machine learning techniques. For example, the primary content may be 'Lake Michigan'. A content provider may be a weather forecasting station close to Lake Michigan. Then, the secondary content may be a weather forecast of the weather in Michigan provided by the weather forecasting station.

[0051] Secondary content may comprise data, e.g., audio data, or video data, or a combination thereof. This data may be configured such that a client device, after having received the data, can process the data into a sequence of video frames that can be displayed on a screen of the client device. For example, secondary content may be data comprised in an mp4 file. Further protocols may be used for formatting the data before transmitting the data to the client device. Examples of protocols comprise HLS, MPEG DASH, RTMP, WebM, mp4, MPEG-TS, and DVB-DASH. as discussed above. The sequence of video frames may provide content to a viewer. The secondary content may be a warning, an alert, an advertisement, or a notification.

[0052] Specific embodiments and further details of inserting secondary content into a media stream are discussed throughout the rest of the description.

[0053] The above embodiment further comprises, as also indicated already above, the step of inserting 105 into the media stream, by content management system 101, and in response to receiving the slot information: announcement information based on the received slot information and to be displayed at a client device, and an interactivity indicator to be displayed at the client device and indicating to a user of the client device that the user, through the client device, can transmit feedback in relation to the announcement information.

[0054] Inserting 105 the announcement information and the interactivity indicator into the media stream may be achieved by various techniques. For example, the media stream may contain, as discussed above, video data, audio data, or a combination thereof. The data may be encoded and transmitted via the internet from a server, for example, content management system 101, to a client device, wherein the data may be decoded at the client device and content corresponding to the data may be displayed at the client device using a media player. The announcement information or the interactivity indicator may be a static image or an animated image that can be stored as a respective static image data file, e.g., a PNG file or a jpeg file, or an animated image data file, e.g., an animated GIF file. Then, while displaying the content generated from processing audio / video data comprised in the media stream, a respective static image or a respective animated image may be displayed alongside (or overlayed over) the content being displayed by a media player of the client device.

[0055] "Based on the received slot information" may mean that the above insertion 105 of the announcement information and interactivity indicating is performed after the slot information has HE Ref.: 269353 been transmitted by secondary content exchange system 102 to content management system 101. Moreover, it may mean that the slot information has been received by content management system 101 before inserting the announcement information and / or interactivity indicator into the media stream. Furthermore, it may mean that the content of the announcement information and / or the content of the interactivity indicator is generated by processing the slot information. For instance, content management system 101 may receive the slot information, process the slot information, and thereby identify a secondary content provided by a content provider (as discussed above).

[0056] Announcement information may comprise graphical elements indicating the secondary content and / or the content provider identified by processing the slot information, thereby allowing a viewer of the graphical elements to identify the secondary content and / or the content provider. An example of the announcement information is a banner that may be inserted into a media stream in the manner discussed above.

[0057] An interactivity indicator may then comprise graphical elements indicating to a user (notifying a user) that the user, through the client device, can transmit feedback in relation to the announcement information. For example, the user may transmit feedback by operating a remote control or any other control interface provided by or connected to the client device. As an example, operating a remote control may involve pressing one button or a combination of buttons on the remote control. As another example, operation a control interface by a client device may comprise performing a gesture on the display of the client device, such as a 'swipe' or a 'tap' on a display of a tablet displaying the interactivity indicator. Further details about transmitting feedback through the client device are discussed below with respect to further embodiments. An example of the interactivity indicator is a colored or otherwise recognizable representation of a button, such as a red square, rectangle, disk, icon, or button, that is inserted into a media stream in the manner discussed above.

[0058] The above embodiment provides, in effect, an advantageous selection mechanism of content to be inserted into a media stream. Specifically, the secondary content to be inserted into the media stream may be selected based on two conditions. Firstly, the secondary content needs to arise from an interest indicated by a content provider based on the secondary content opportunity information that is determined based on the content of a media stream. This aspect is reflected by the abovediscussed considerations on secondary content opportunity information and slot information, and their interdependency. Secondly, the secondary content needs to have been approved based on feedback by a user. This aspect is reflected by the above-discussed considerations on inserting announcement information and an interactivity indicator. This prevents the displaying of secondary content that would otherwise be displayed without the above selection mechanism. This reduces the burden on the network and the burden on the client device. Thereby, processing resources and energy can be saved. Saving energy is particularly beneficial for battery-powered devices. HE Ref.: 269353

[0059] The method according to the above embodiment may be computer-implemented. There may be a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of the above embodiment. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of the above embodiment.

[0060] The above embodiment or any of its parts may be combined with any other embodiment or any parts of another embodiment as discussed throughout this description.

[0061] Figure 2 shows an example of a second embodiment of the invention, wherein the secondary content opportunity information indicates a plurality of associations between a primary content and a temporal reference.

[0062] As noted above, the secondary content opportunity information may refer to a plurality of bits, wherein, if the bits are processed by a computer, the computer may be able to present to a user via a display of the computer a graphical representation of the information. Figure 2 shows an example of this graphical representation of the secondary content opportunity information, wherein the secondary content opportunity information indicates a plurality of associations.

[0063] Here, the column on the left hand side contains examples of instances of primary content, such as the words 'Venice', 'Lake Michigan', 'Mount Everest', 'Rain'. Moreover, primary content may also comprise a plurality of words, such as 'Sunshine' and 'Water'.

[0064] The column on the right-hand side contains examples of temporal references. For instance, a temporal reference may be a time span denoted in the format hh:mm:ss - hh:mm:ss, wherein hh denotes hours, mm denotes minutes, and ss denotes seconds, and the hyphen or dash is used in representing a range beginning at the first time instance and ending at the second instance. With respect to the hyphen or dash, hh:mm:ss - hh:mm:ss may be expressed as 'from hh:mm:ss to hh:mm:ss'. In another case, a temporal reference may be a time step, for example: 01:12:21, which denotes a time stamp generated for hours equal to 1, minutes equal to 12, and seconds equal to 21. In another case, a temporal reference may be a time window. A time window may be a specific time span that is labelled. For example, the time span 02:30:12 - 02:31:01 may be labelled as Window 3. A temporal reference may be a combination of any or all of the above examples. This is shown in Figure 2 as Window 8; 00:34:12 - 00:40:00; 01:50:00, where a temporal reference comprises time Window 8, a time span ranging from 00:34:12 to 00:40:00, and a time stamp at 01:50:00. Figure 2 shows another example of a temporal reference comprising a combination of a time window and a time stamp, namely a time window Window 1 and a time stamp at 00:06:06. HE Ref.: 269353

[0065] Further details on the concept of associations are discussed above with respect to embodiment 1 depicted in Figure 1. In brief, one example of an association is a relation between a primary content and a temporal reference. Figure 2 depicts an example of a plurality of associations. As can be seen from Figure 2 there is an association between the primary content 'Venice' and a temporal reference '00:00:00 - 00:05:01'. Moreover, there is an association between the primary content 'Lake Michigan' and a temporal reference '01:12:21'. Corresponding explanations apply to the further instances of primary content denoted in the column on the left-hand side.

[0066] Further, the table as shown in Figure 2 depicts an example of a one-to-one, 1:1, relationship between primary content and temporal reference, namely (Lake Michigan): (01:12:21). The table furthermore depicts an example of a 1-to-many, l:n, relationship between primary content and temporal reference, namely (Rain) : (Window 8; 00:34:12 - 00:40:00; 01:50:00). Moreover, the table depicts an example of a many-to-one, n:l, relationship between primary content and temporal reference, namely (Mount Everest; Snow): (Window 3). Furthermore, the table depicts an example of a many- to-many, n:m, relationship between primary content and temporal reference, namely (Sunshine; Water):(Window 1; 00:06:06). Further details on the concept of relationships and an association are discussed with respect to the first embodiment and Figure 1.

[0067] As mentioned above, one of the two conditions of the advantageous selection mechanism is that the secondary content needs to arise from an interest indicated by a content provider based on the secondary content opportunity information that is determined based on the content of a media stream. According to the present embodiment the secondary content opportunity information indicates a plurality of associations between a primary content and a temporal reference. Thereby, the above advantageous according to the first embodiment can be applied to a plurality of temporal references at which primary content is to be shown in the media stream. E.g., the selection mechanism may not only be applied to a scene showing Venice, but also to another scene showing Lake Michigan, and to other scenes showing Rain. This prevents the displaying of secondary content that would otherwise be displayed at multiple time instances throughout audiovisual media displayed in the media stream. This reduces the burden on the network and the burden on the client device even further. Moreover, even more processing resources and energy can be saved.

[0068] The above embodiment or any of its parts may be combined with any other embodiment or any parts of another embodiment as discussed throughout this description.

[0069] Figure 3 shows a diagram depicting an example of a third embodiment of the invention, wherein method of any one of the preceding embodiments further comprises the step of: before sending the secondary content opportunity information 103 by content management system 101 to secondary content exchange system 102, generating 106, by content management system 101, the secondary content opportunity information 103. HE Ref.: 269353

[0070] As also shown in the diagram, the step of generating 106 secondary content opportunity information is performed before sending the secondary opportunity information 103. In other words, the step of sending the secondary opportunity information 103 may be performed after the secondary content opportunity information was generated in step 106. There is no limit to a temporal distance or time span between the step of generating 106 and the step of sending 103. For example, first, the step of generating 106 may be performed, and then, after a time span of 5 hours has lapsed, the step of sending 103 may be performed. As another example, the temporal distance between generating 106 and sending 103 may be severely days, or even a few minutes or seconds.

[0071] Generating secondary content opportunity information may also be referred to as producing, computing, calculating, or creating secondary content opportunity information. In general, generating data may comprise processing input data, for example, the data included in the media stream, by a computer, thereby computing the data, for example, the secondary content opportunity information. A specific example of the generating is discussed as a following embodiment. As outlined above, the mentioned computer that may process input data may be content management system 101.

[0072] The content management system 101 generating the secondary content opportunity information 106 means that the other constituents of the networks, for example, the secondary content exchange system 102, do not need to process the data included in the media stream. Moreover, the data included in the media stream do not need to be transmitted for generating the secondary content opportunity information. This further reduces the burden on the network. Thereby, even more processing resources and energy can be saved.

[0073] The above embodiment or any of its parts may be combined with any other embodiment or any parts of another embodiment as discussed throughout this description.

[0074] Figure 4 shows a diagram depicting an example of a fourth embodiment of the invention, wherein the step of generating the secondary content opportunity information 106, as discussed with respect to the third embodiment, comprises identifying 107, in at least one part of the media stream, the primary content at the temporal reference at which the primary content appears in the part of the media stream, by using a machine learning based process, generating 108 the association between the identified primary content and the temporal reference, and generating 109 the secondary content opportunity based on the generated association.

[0075] Identifying 107 primary content by using a machine learning based process may mean that content management system 101 executes a machine learning method, thereby identifying the primary content. Identifying 107 the primary content may mean determining the primary content that HE Ref.: 269353 corresponds to the part of the media stream. Executing the machine learning method may comprise inputting input data to the machine learning method, automatically analyzing the input data by the machine learning method, and outputting the result of the analyzing.

[0076] Examples of a machine learning method are a linear regression supervised learning method, a K-means clustering unsupervised learning method, or a random forest ensemble learning method. A machine learning method comprises an artificial intelligence method.

[0077] The machine learning method may be implemented as a computer program comprising instructions which, when executed by a computer or content management system 101, cause the computer or content management system 101 to execute the above steps of inputting, automatically analyzing, and outputting in the above manner.

[0078] The input data may be the part of the media stream, or the data, audio data, video data, or audio and video data comprised in the part of the media stream.

[0079] Inputting input data into the machine learning method may include loading the input data into a computer's processing memory, or the content management system's processing memory. Examples of a processing memory are a random access memory (RAM), a cache, or a register of a central processing unit (CPU).

[0080] Analyzing the input data by the machine learning method may include the CPU of content management system 101 executing the machine learning method, whereby the input data that was loaded into the processing memory is processed, thereby computing output data. The output data may indicate a primary content of the part of the media stream. For example, the machine learning method may determine an object that is depicted in the video frames comprised in the media stream. The output data may indicate the determined object.

[0081] The clause "part" comprises a fraction, a piece, or a share, or any combination thereof, this applying to the entire disclosure of the present document. As discussed above, the media stream may comprise data. The part of the media stream may be a part of the data. For example, the media stream may comprise audio and video data that, if played back by a media player, would correspond to a fixed amount of time, for example, 100 minutes of playback data. Now, a part of the media stream may correspond to a part of this fixed amount of time, for example 10 minutes, or piece, for example, one third, 33 minutes of the fixed amount of time.

[0082] Generating 108 the association between the identified primary content and the temporal reference may mean computing data that, if processed by a computer, indicates a relationship between primary content and a temporal reference. The association may be the association as discussed HE Ref.: 269353 throughout the above embodiments. The term generating may comprise the same meaning and concepts as discussing throughout the above embodiments. For instance, a computer, such as content management system 101, may compute data corresponding to an association as shown as one of the rows of the table depicted in Figure 2.

[0083] Generating 109 the secondary content opportunity information based on the generated association may comprise processing the association generated in step 108, for example, converting the data representing the association into a data that has a format such that it can be transmitted to another device, for example, secondary content exchange system 102.

[0084] As an advantage of the above embodiment, identifying primary content by using a machine learning based process allows for determining the primary content with higher determination accuracy. In comparison, techniques different from machine learning may be used for determining the primary content. However, these techniques usually result in a lot of erroneous results. In comparison thereto, machine learning provides for less error prone results. Thereby, repeated calculations of primary content, which are due to erroneous computations based on other techniques, can be omitted. Thereby, energy can be saved.

[0085] The above embodiment or any of its parts may be combined with any other embodiment or any parts of another embodiment as discussed throughout this description.

[0086] Figure 5 shows a diagram depicting an example of a fifth embodiment of the invention, wherein the method of any one of the preceding embodiments further comprises generating 110, by secondary content exchange system 102, in response to receiving 103 the secondary content opportunity information, the slot information, and sending 104, by secondary content exchange system 102, to content management system 101, the slot information.

[0087] Generating slot information has been discussed in detail throughout the above embodiments and comprises corresponding details. Further details of generating slot information are provided in the following embodiments.

[0088] In other words, generating 110 slot information in response to receiving the secondary content opportunity information may mean that the step of generating 110 the slot information may be performed by secondary content exchange system 102 after secondary content exchange system 102 has received 103 the secondary content opportunity information. This is indicated by the arrows labelled 103 and 104 in Figure 5. Here, arrow 103 depicts that secondary content opportunity information is sent by content management device 101 to secondary content exchange system 102. In other words, arrow 103 shows that secondary content exchange system 102 receives the secondary content opportunity information from content management system 101. Corresponding HE Ref.: 269353 concepts apply to the arrow 104. Now, after receiving the secondary content opportunity information, secondary content exchange system 102 performs the step of generating the slot information. Slot information 110 may be generated based on real-time update information. The updated information may be generated based on a bidding process. The bidding process may involve a plurality of providers of secondary content, wherein the providers are interested in inserting their respective secondary content, and may make bids for inserting their respective secondary content. Therein, for example, the provider with the highest bid may obtain the opportunity to insert the secondary content.

[0089] The specifics of slot information as mentioned in this embodiment are discussed throughout the above embodiments.

[0090] Sending 104, by secondary content exchange system 102, to content management system 101, the slot information may mean that secondary content exchange system 102 may transmit a message, for example, a request message of the HTTPS PUSH method as discussed throughout the above embodiment, to content management system 101, wherein content management system 101 may receive this message in a manner as discussed throughout the above embodiments.

[0091] As an advantage, generating the slot information in response to receiving the secondary content opportunity information at the secondary content exchange system results in reduced transmissions in the complete system. This is because no further signaling is required for generating the slot information.

[0092] The above embodiment or any of its parts may be combined with any other embodiment or any parts of another embodiment as discussed throughout this description.

[0093] A sixth embodiment provides a method according to the fifth embodiment, wherein generating the slot information comprises determining the interest in inserting the secondary content based on the secondary content opportunity information.

[0094] Determining the interest in inserting the secondary content based on the secondary content opportunity information may mean determining data which, if the data is processed by content management system 101, identifies a secondary content provided by a content provider, wherein the content provider may be interested in having the secondary content inserted into the media stream.

[0095] As an advantage, generating the slot information based on the secondary content opportunity information allows for an automated way of generating the contents of the slot information. Further computations or transmissions, for instance, of requests to databases requesting further data HE Ref.: 269353 needed to generate the slot information, are not needed. Hence, transmissions are reduced.

[0096] Thereby, energy consumption is reduced.

[0097] The above embodiment or any of its parts may be combined with any other embodiment or any parts of another embodiment as discussed throughout this description.

[0098] A seventh embodiment provides a method of any one of the preceding embodiments further comprising inserting the secondary content into the media stream in response to the user transmitting the feedback.

[0099] Further details about inserting the secondary content into the media stream will be discussed in following embodiments.

[0100] Feedback may mean an indication that the user is interested in seeing the secondary content. In other words, the feedback may indicate that the secondary content shall be inserted into the media stream. Further details about a user transmitting feedback will be discussed in following embodiments.

[0101] A user may be a person interacting with the client device or operating the client device.

[0102] Inserting the secondary content in response to the user transmitting the feedback may mean that the feedback is transmitted before the secondary content is inserted. Moreover, it may mean that transmitting the feedback may trigger, or initiate, the inserting of the secondary content, for example, by any of the approaches discussed throughout the above or following embodiments.

[0103] As an advantage, inserting the secondary content in response to a user transmitting feedback provides a simplified condition for inserting the secondary content. For example, in other cases, it may be determined whether secondary content may be inserted based on elaborate mechanisms including various parameters. Using the above approach, these may, in some embodiments, be made superfluous. Thereby, processing steps are saved. Thereby, energy is saved.

[0104] The above embodiment or any of its parts may be combined with any other embodiment or any parts of another embodiment as discussed throughout this description.

[0105] Figure 6 shows a diagram depicting aspects of an eighth embodiment of the invention. The eighth embodiment provides a method according to the seventh embodiment wherein inserting the secondary content comprises receiving, by the client device 111, the feedback 112 from the user 113, and sending, by client device 111, to a third processing system 114, hereinafter referred to as HE Ref.: 269 353

[0106] "secondary content stitcher", information 115, hereinafter referred to as "feedback information", indicating the feedback of user 113.

[0107] Receiving, by client device 111, feedback 112 from user 113 may mean that client device 111 by various means receives a user's feedback. An example of one of the various means may be a message transmitted from a remote device to client device 111. An example of a remote device may be a remote control. An example of client device 111 may be a smart TV. The message transmitted may be a message transmitted by wireless communications. Examples of wireless communications here comprise infrared or radiofrequency transmission. Another example of a remote device may be smartphone that is wirelessly connected to client device 111, for example by Bluetooth, or via a wireless local area network.

[0108] Secondary content stitcher 114 may be a computer or a server. Secondary content stitcher 114 may act as an intermediary device between secondary content exchange system 102 and client device 111. An intermediary device may be a device configured to forward messages from secondary content exchange system 102 to client device 111, and vice versa. Secondary content stitcher 114 may be further configured to compute data. Further aspects of secondary content stitcher 114 are discussed in the following embodiments.

[0109] As indicated by the time arrow shown in Figure 6, the sequence of first receiving the feedback 112 and then sending the feedback information 115 may be performed after the announcement information and the interactivity indicator have been inserted into the media stream, and after both the announcement information and the interactivity indicator have been displayed at a display of the client device, thereby allowing user 113 of client device 111 to notice that user 113 may transmit the feedback through client device 111.

[0110] As an advantage, sending the feedback to the secondary content stitcher instead of the content management system reduces the computational burden at the content management system and thereby may save power, and also may reduce storage that may be necessary for computations at the content management system. Thereby, energy may be saved at the content management system.

[0111] The above embodiment or any of its parts may be combined with any other embodiment or any parts of another embodiment as discussed throughout this description.

[0112] Figure 7 shows a diagram depicting an example of a ninth embodiment of the invention. The ninth embodiment provides a method according to the eighth embodiment wherein inserting the secondary content further comprises sending, by secondary content stitcher 114, to secondary HE Ref.: 269353 content exchange system 102, forwarded feedback information 116 indicating the feedback of user 113.

[0113] As indicated in Fig. 7 , the step of sending forwarded feedback information 116 may be performed after receiving feedback information 115 by secondary content stitcher 114. Receipt of feedback information 115 may be a condition for starting the sending or transmitting of forwarded feedback information 116. Forwarded feedback information 116 may be the same information as feedback information 115. Forwarded feedback information 116 may be a processed feedback information 115. For instance, secondary content stitcher 114 may receive feedback information 115, process feedback information 115, and thereby compute forwarded feedback information 116. An example of computing forwarded feedback information 116 is converting feedback information 115 into another format.

[0114] Forwarded feedback information 116 may be sent from secondary content stitcher 114 to secondary content exchange system 102 using methods comprised in technical specifications such as for example provided by the Interactive Advertising Bureau (IAB). An example of such specification is the Video Ad Serving Template (VAST) protocol for linear channel TV. Another example of such specification is the Video Multiple Ad Playlist (VMAP) protocol for video on demand (VOD) content.

[0115] As an advantage, sending the feedback from the secondary content stitcher to the secondary content exchange system 102, instead of forwarding it via the content management system, reduces the computational burden at the content management system and thereby may save power, and also may reduce the storage that may be necessary for computations at the content management system. Thereby, energy may be saved at the content management system.

[0116] The above embodiment or any of its parts may be combined with any other embodiment or any parts of another embodiment as discussed throughout this description.

[0117] Figure 8 shows a diagram depicting an example of a tenth embodiment of the invention. The tenth embodiment provides a method according to the ninth embodiment wherein inserting the secondary content comprises determining 117, by secondary content exchange system 102, based on received forwarded feedback information 116, the secondary content to be inserted, and sending, by secondary content exchange system 102, to secondary content stitcher 114, secondary content reference information 118 indicating the secondary content to be inserted.

[0118] Secondary content to be inserted may mean secondary content that shall be inserted into the media stream delivered to client device 111. HE Ref.: 269353

[0119] Determining the secondary content to be inserted based on the received forwarded feedback information may mean that secondary content exchange system 102 processes forwarded feedback information 116 and thereby determines a secondary content to be inserted. For example, forwarded feedback information 116 may indicate the feedback of user 113, wherein the feedback of user 113 indicates that user 113 is interested in seeing one out of a plurality of secondary content instances that may be inserted into the media stream. Secondary content exchange system 102 may determine that the secondary content to be inserted is the one out of the plurality of secondary content instances.

[0120] Secondary content reference information 118 indicating the secondary content to be inserted may be data that, if processed by secondary content stitcher 114, indicates, to secondary content stitcher 114, the secondary content to be inserted. In other words, by processing secondary content reference information 118, secondary content stitcher 114 may determine the secondary content to be inserted. An indication of the secondary content to be inserted may comprise a pointer, an index, or an identifier of the secondary content. For example, the indication may be one or a plurality of bits.

[0121] Sending secondary content reference information 118 may mean that a message is transmitted by secondary content exchange system 102 to secondary content stitcher 114. This message may comprise secondary content reference information 118. For example, the message may comprise a data field that comprises secondary content reference information 118. The message may be transmitted using a wireless or wired connection. Further details and examples of messages and connection types used for the transmission are discussed throughout the above and below embodiments.

[0122] As an advantage, sending the feedback from the secondary content stitcher to the secondary content exchange system 102, instead of forwarding it via the content management system, reduces the computational burden at the content management system and thereby may save power, and also may reduce the storage that may be necessary for computations at the content management system. Thereby, energy may be saved at the content management system.

[0123] The above embodiment or any of its parts may be combined with any other embodiment or any parts of another embodiment as discussed throughout this description.

[0124] Figure 9 shows a diagram depicting an example of an eleventh embodiment of the invention. The eleventh embodiment provides a method according to the tenth embodiment wherein inserting the secondary content further comprises receiving, by secondary content stitcher 114, secondary content reference information 118, computing 119, by secondary content stitcher 114, a configuration file 120, based on which client device 111 can insert the secondary content into the HE Ref.: 269353 media stream, and sending, by secondary content stitcher 114, to client device 111, configuration file

[0125] 120.

[0126] Configuration file 120 may be data that, if processed by client device 111, indicates to client device 111 another device or a location from which client device 111 can retrieve the media stream and the secondary content. For example, configuration file 120 may comprise an indicator of a server, wherein the server may store the media stream and / or the secondary content. The indicator may identify a resource stored at the server. The resource may be the media stream and / or the secondary content. An example of the indicator is a uniform resource identifier (URI) or uniform resource locator (URL). An example of configuration file is a manifest file. Examples of a manifest file are ,m3u8 files used for streaming based on the HLS protocol, .mpd files used for streaming based on the MPEG-DASH protocol, and .ismc files used for streaming based on Microsoft smooth streaming.

[0127] Computing configuration file 120 may mean that secondary content stitcher 114 computes configuration file 120 based on the received secondary content reference information 118.

[0128] Sending configuration file 120 to client device 111 may be done in line with any of the approaches discussed herein throughout the above and below embodiments. Moreover, for example, configuration file 120 may be comprised in a message transmitted by secondary content stitcher 114 to client device 111 using a wireless connection and may involve using any communications protocol, for example, HTTPS or TCP / IP.

[0129] As an advantage, the above embodiment allows for inserting secondary content using a modified configuration file, based on which client device 111 can insert the secondary content into the media stream. Thereby, the secondary content to be inserted does not need to be transmitted from secondary content exchange system 102, content management system 101, or secondary content stitcher 114. Instead, client device 111 can request the secondary content directly from an external device based on the configuration file. Thereby, transmissions in the system comprising content management system 101, secondary content exchange system 102 and secondary content stitcher 114, are reduced. This reduces the computational burden on the system. Moreover, energy is saved.

[0130] The above embodiment or any of its parts may be combined with any other embodiment or any parts of another embodiment as discussed throughout this description.

[0131] Figure 10 shows a diagram depicting an example of a twelfth embodiment of the invention. The twelfth embodiment provides a method according to the eleventh embodiment wherein inserting the secondary content further comprises inserting 121, by client device 111, based on configuration file 120, the secondary content into the media stream. HE Ref.: 269353

[0132] Inserting 121 secondary content into the media stream may comprise determining, based on configuration file 120, a locator of the secondary content. The secondary content may be stored on an external device, for example, a streaming server. The locator may be an address of the secondary content. The address may be a URI or URL. Furthermore, inserting 121 may comprise retrieving the secondary content based on the locator. Retrieving the secondary content may involve sending, by client device 111, a request to the external device. The request may indicate that the external device shall send the secondary content to client device 111. The request may be a HTTPS request. The external device may be determined by the locator. For example, a streaming server may be determined by a URL. After the external device has received the request, the external device may send a response message to the client device 111 that has sent the request. The response message may comprise the secondary content. Client device 111 may insert or integrate or stitch the received secondary content into the media stream.

[0133] The integrating, inserting, or stitching may involve determining two markers comprised in the media stream. A first marker may determine a first temporal reference in the media stream. A second marker may determine a second temporal reference in the media stream. An example of a marker is an SCTE-35 marker. Another example of a marker is a cue point. For example, the first marker be a cue-in point and the second marker be a cue-out point. SCTE markers may be used for linear channels. CUE points may be used for VOD content.

[0134] Client device 111 may start displaying the secondary content and stop displaying the media stream based on the first temporal reference. After the secondary content has been displayed, client device 111 may restart displaying the media stream based on the second temporal reference. The first temporal reference may correspond to a first video frame. The second temporal reference may correspond to a second video frame. The second video frame may be a video frame following the first video frame.

[0135] For example, displaying the media stream may be stopped at the first video frame and restarted at the second video frame. The computing of a respective video frame may be based on the video frame rate and the temporal reference. For example, the first marker may determine the first temporal reference as 10:00, i.e., minutes = 10, seconds = 00. The temporal reference of 10:00 may correspond to the number of seconds corresponding to 10 minutes times the frame rate, frames per second (fps). Thereby, an index or a number linked to a video frame shown at 10:00 may be computed. For example, the temporal reference may correspond to 600 seconds times 30 fps, which equals to 18000 video frames. The number of the first video frame may thus be 18000. The second video frame then may be 18001. Client device 111 may stop displaying the media stream, for example, a movie, or news, at minute 10 and seconds 00, or video frame 18000. Then, after the secondary content, for example, a weather forecast, has been displayed, client device 111 may HE Ref.: 269353 restart displaying the media stream at the second temporal reference, for example at video frame

[0136] 18001.

[0137] As an advantage, the approach of the above embodiment, in line with the above considerations on the eleventh embodiment, reduces transmissions and thereby saves energy.

[0138] The above embodiment or any of its parts may be combined with any other embodiment or any parts of another embodiment as discussed throughout this description.

[0139] Figure 11 shows a diagram depicting an example of a thirteenth embodiment of the invention. The thirteenth embodiment provides a method according to any one of the preceding embodiments, further comprising, after receiving the slot information, setting 122, by content management system 101, based on the slot information, the temporal reference at which the secondary content is to be inserted into the media stream.

[0140] Here, the slot information may comprise additional data indicating a specific temporal reference at which the secondary content shall be inserted into the media stream. The additional data may indicate at least one specific temporal reference, or a plurality of specific temporal references.

[0141] The media stream may comprise at least one default temporal reference, or a plurality of default temporal references. A default temporal reference may be a temporal reference at which secondary content is to be inserted into the media stream by default. For example, a default temporal reference may be positioned at the end of one part of the media stream. For example, the one part of the media stream may correspond to a part of a movie displayed by a media player executed by client device 111. The part of the movie may be a scene of the movie, for example, a scene playing at Lake Michigan.

[0142] Here, a specific temporal reference as mentioned above may differ from the default temporal reference. For example, a default temporal reference may be 00:10, using the notation as introduced in the above embodiments, and the specific temporal reference may be 01:00, using the notation as introduced in the above embodiments.

[0143] By processing the additional data, content management system 101 may set the temporal reference at which the secondary content is to be inserted into the media stream to the specific temporal reference. In other words, setting the temporal reference may mean that content management system 101 may replace default temporal reference with the specific temporal reference. For example, the additional data comprised in the slot information may indicate 01:00. Then, content management system 101 may set the temporal reference at which the secondary content is to be inserted to 01:00. HE Ref.: 269353

[0144] Moreover, setting the temporal reference may comprise, by content management system 101, adding an additional temporal reference at which the secondary content is to be inserted into the media stream to the media stream, wherein the additional temporal reference is the specific temporal reference.

[0145] As an advantage, the approach of the above embodiment allows for automatically rearranging, adding, or removing temporal references at which secondary content may be inserted into a media stream. Rearranging, adding, or removing temporal reference may be achieved by other, more computation and transmissions-heavy approaches. Thus, the simplified approach according to the above embodiment, reduces the computational burden on the system and saves energy.

[0146] The above embodiment or any of its parts may be combined with any other embodiment or any parts of another embodiment as discussed throughout this description.

[0147] Figure 12a shows a diagram depicting an example of a fourteenth embodiment of the invention. The fourteenth embodiment provides a first processing system 200, here referred to as "content management system", for preparing content of a media stream, the content management system 200 being configured for: sending, by the content management system 200, to a second processing system here referred to as "secondary content exchange system", information, here referred to as "secondary content opportunity information", wherein the secondary content opportunity information indicates an association between content, here referred to as "primary content", appearing in the media stream and a temporal reference at which the primary content appears in the media stream; receiving, by the content management system 200 from the secondary content exchange system, in response to sending the secondary content opportunity information, information, here referred to as "slot information", indicating an interest in inserting secondary content into the media stream; and inserting into the media stream, in response to receiving the slot information: information, here referred to as "announcement information", based on the received slot information and to be displayed at a client device, and an indicator, here referred to as "interactivity indicator", to be displayed at the client device, the interactivity indicator indicating to a user of the client device that the user, through the client device, can transmit feedback in relation to the announcement information.

[0148] The content management system 200, secondary content exchange system, and client device may, respectively, be a content management system, content exchange system, and client device as discussed throughout the above embodiments.

[0149] As a non-limiting example, sending may be carried out by a sending means 201, a sending unit 201, or a sending module 201, each of which individually may be comprised by the content management HE Ref.: 269353 system 200, and each of which individually may be configured to send the secondary content opportunity information. Herein, a means or a unit may be a specific part of computer hardware configured for sending messages, for example, via the internet to another computer. For example, a sending means 201 may be a network interface card. A sending module may be a software implementation to be carried by a computer which instructs a computer to perform an operation of sending based on the computer's hardware configuration.

[0150] As a non-limiting example, receiving may be carried out by a receiving means 202, a receiving unit

[0151] 202, or a receiving module 202, each of which individually may be comprised by the content management system 200, and each of which individually may be configured to receive the slot information. Herein, a means or a unit may be a specific part of computer hardware configured for receiving messages, for example, via the internet from another computer. For example, a receiving means may be a network interface card. A receiving module 202 may be a software implementation to be carried by a computer which instructs a computer to perform an operation of receiving based on the computer's hardware configuration.

[0152] As a non-limiting example, inserting may be carried out by an inserting means 203, an inserting unit

[0153] 203, or an inserting module 203, each of which individually may be comprised by the content management system 200, and each of which individually may be configured to insert announcement information and an interactivity indicator. Herein, a means or a unit may be a specific part of computer hardware configured for inserting information into a media stream. For example, an inserting means 203 may be a network interface card. An inserting module may be a software implementation to be carried by a computer which instructs a computer to perform an operation of inserting based on the computer's hardware configuration.

[0154] The sending means 201, receiving means 202, or inserting means 203 may each be a part of computer hardware that is specifically designed to carry out the specified functions. For example, the sending means 201 may be a field programmable gate array (FPGA) that is built to carry out the above operations. The above sending means 201, receiving means 202, or inserting means 203 may instead be implemented as software that may be executed by the content management system 200.

[0155] At least for the reasons discussed with respect to the first embodiment, the present embodiment, an advantage, may be saving processing resources and energy, wherein saving energy may be particularly beneficial for battery-powered devices.

[0156] The above embodiment or any of its parts may be combined with any other embodiment or any parts of another embodiment as discussed throughout this description. HE Ref.: 269353

[0157] Figure 12b shows a diagram depicting an example of a fifteenth embodiment of the invention. The fifteenth embodiment provides a content management system 200 of the fourteenth embodiment wherein the content management system is further configured for: setting, based on the slot information, a temporal reference at which the secondary content is to be inserted into the media stream.

[0158] As a non-limiting example, setting may be carried out by a setting means 204, a setting unit, or a setting module, each of which individually may be comprised by the content management system 200, and each of which individually may be configured to set a temporal reference at which the secondary content is to be inserted into the media stream. Herein, a means or a unit may be a specific part of computer hardware configured for setting temporal references, for example, by communicating with other computer or servers via the internet. For example, an inserting means may be a network interface card. An inserting module may be a software implementation to be carried by a computer which instructs a computer to perform an operation of setting a temporal reference based on the computer's hardware configuration.

[0159] The above embodiment or any of its parts may be combined with any other embodiment or any parts of another embodiment as discussed throughout this description.

[0160] Concepts, background, and possible advantages of this embodiment can be found throughout the above embodiments, for example, the thirteenth embodiment.

[0161] Figure 13a shows a diagram depicting an example of a sixteenth embodiment of the invention. The sixteenth embodiment provides a system 300 comprising: a first processing system, here referred to as a "content management system" 301, a second processing system, here referred to as "secondary content exchange system" 302, a client device 303, and a third processing system, here referred to as "secondary content stitcher" 304; wherein the content management system 301 is configured for sending 305, by the content management system 301, to the secondary content exchange system 302, information, here referred to as "secondary content opportunity information", wherein the secondary content opportunity information indicates an association between content, here referred to as "primary content", appearing in the media stream, and a temporal reference at which the primary content appears in the media stream; receiving 306, from the secondary content exchange system 302, in response to sending the secondary content opportunity information, information here referred to as "slot information", indicating an interest in inserting content, here referred to as "secondary content", into the media stream; and inserting 307 into the media stream, by the content management system 301, and in response to receiving the slot information: information, here referred to as "announcement information", based on the received slot information, and to be displayed at the client device 303, and an indicator, here referred to as "interactivity indicator", to be displayed at the client device 303, the interactivity indicator indicating to a user of the client device HE Ref.: 269353

[0162] 303 that the user, through the client device 303, can transmit feedback in relation to the announcement information.

[0163] The content management system 301, secondary content exchange system 302, client device 302, and secondary content stitcher 304 may, respectively, be a content management system, content exchange system, client device and secondary content stitcher as discussed throughout the above embodiments.

[0164] For example, the operations of sending, receiving, inserting as discussed above may be the operations of the fourteenth embodiment.

[0165] At least for the reasons discussed with respect to the first embodiment, the present embodiment, as an advantage, may allow for saving processing resources and energy, wherein saving energy may be particularly beneficial for battery-powered devices.

[0166] The above embodiment or any of its parts may be combined with any other embodiment or any parts of another embodiment as discussed throughout this description.

[0167] Figure 13b shows a diagram depicting an example of a seventeenth embodiment of the invention. The seventeenth embodiment provides a system of the sixteenth embodiment wherein: the content management system 301 is configured for, before sending 305 the secondary content opportunity information by the content management system 301 to the secondary content exchange system 302: generating the secondary content opportunity information; the secondary content exchange system 302 is configured for generating, in response to receiving the secondary content opportunity information, the slot information, and sending 306, to the content management system 301, the slot information; the client device 303 is configured for receiving the feedback from the user, and sending 308 to a third processing system, hereinafter referred to as "secondary content stitcher" 304, information, hereinafter referred to as "feedback information", indicating the feedback of the user; the secondary content stitcher 304 is configured for sending 309, to the secondary content exchange system 302, information, hereinafter referred to as "forwarded feedback information", indicating the feedback of the user; the secondary content exchange system 302 is configured for determining, by the secondary content exchange system, based on the received forwarded feedback information, the secondary content to be inserted, and sending 310, to the secondary content stitcher 304, information, hereinafter referred to as "secondary content reference information", indicating the secondary content to be inserted; the secondary content stitcher 304 is configured for receiving, by the secondary content stitcher 304, the secondary content reference information, and computing, by the secondary content stitcher 304, information, hereinafter referred to as "configuration file", based on which the client device 303 can insert the secondary content into the media stream, and sending 311, by the secondary content stitcher 304, to the client device 303, the configuration file. HE Ref.: 269353

[0168] As a non-limiting example, the operations of sending, receiving, and inserting may be carried out by the means, units, or modules. These may, respectively, be comprised in respective processing system, for example, the content management system 301, the secondary content exchange system 302, the client device 303, or the secondary content stitcher 304. Further details are discussed in above embodiments.

[0169] As a non-limiting example, generating may be carried out by a generating means, a generating unit, or a generating module, each of which individually may be comprised by the content management system 301, the secondary content exchange system 302, the client device 303, or the secondary content stitcher 304. Each of which may be configured to generate information of the type as discussed above. Herein, a means or a unit may be a specific part of computer hardware configured for generating information, for example, by processing input data into output data by a central processing unit (CPU). A generating module may be a software implementation to be carried by a computer which comprises instructions that when carried out by a computer cause the computer to perform an operation of generating the above-mentioned types of information in the manner discussed above.

[0170] As a non-limiting example, determining may be carried out by a determining means, a determining unit, or a determining module, each of which individually may be comprised by the content management system 301, the secondary content exchange system 302, the client device 303, or the secondary content stitcher 304. Each of which may be configured to determine the secondary content to be inserted as discussed above. Herein, a means or a unit may be a specific part of computer hardware configured for determining secondary content, for example, by processing input data into output data by a central processing unit (CPU). A determining module may be a software implementation to be carried by a computer which comprises instructions that when carried out by a computer cause the computer to perform an operation of determining the secondary content in the manner discussed above.

[0171] As a non-limiting example, computing may be carried out by a computing means, a computing unit, or a computing module, each of which individually may be comprised by the content management system 301, the secondary content exchange system 302, the client device 303, or the secondary content stitcher 304. Each of which may be configured to compute the configuration file as discussed above. Herein, a means or a unit may be a specific part of computer hardware configured for computing a configuration file, for example, by processing input data into output data by a central processing unit (CPU). A computing module may be a software implementation to be carried by a computer which comprises instructions that when carried out by a computer cause the computer to perform an operation of computing a configuration file in the manner discussed above. 1 HE Ref.: 269353

[0172] Details and possible advantages of this embodiment are discussed throughout the above embodiments.

[0173] The above embodiment or any of its parts may be combined with any other embodiment or any parts of another embodiment as discussed throughout this description.

[0174] The devices and devices comprised in the system as discussed in the above embodiments are nonlimiting examples according to the present invention. The devices and devices comprised in the system according to the above embodiments may either be configured for carrying out, or may further comprise means, units or modules which may be configured to carry out any of the steps comprised in the methods according to the above embodiments. For instance, the content management system according to the fourteenth embodiment may be, in line with the above third embodiment, configured for, before sending the secondary content opportunity information by the content management system to the secondary content exchange system: generating, by the content management system, the secondary content opportunity information.

Claims

HE Ref.: 269353Claims1. A method for preparing content of a media stream, the method comprising: sending, by a first processing system, hereinafter referred to as "content management system", to a second processing system, hereinafter referred to as "secondary content exchange system", information, hereinafter referred to as "secondary content opportunity information", wherein the secondary content opportunity information indicates an association between content, hereinafter referred to as "primary content", appearing in the media stream, and a temporal reference at which the primary content appears in the media stream; receiving, by the content management system from the secondary content exchange system, in response to sending the secondary content opportunity information, information, hereinafter referred to as "slot information", indicating an interest in inserting content, hereinafter referred to as "secondary content", into the media stream; and inserting into the media stream, by the content management system, and in response to receiving the slot information: information, hereinafter referred to as "announcement information", based on the received slot information and to be displayed at a client device, and an indicator, hereinafter referred to as "interactivity indicator", to be displayed at the client device, the interactivity indicator indicating to a user of the client device that the user, through the client device, can transmit feedback in relation to the announcement information.

2. The method of claim 1, wherein the secondary content opportunity information indicates a plurality of associations between a primary content and a temporal reference.

3. The method of any one of the preceding claims, further comprising, before sending the secondary content opportunity information by the content management system to the secondary content exchange system: generating, by the content management system, the secondary content opportunity information.

4. The method of claim 3, wherein generating the secondary content opportunity information comprises identifying, in at least one part of the media stream, the primary content at the temporal reference at which the primary content appears in the part of the media stream, by using a machine learning based process; generating the association between the identified primary content and the temporal reference; andHE Ref.: 269353 generating the secondary content opportunity information based on the generated association.

5. The method of any one of the preceding claims, further comprising generating, by the secondary content exchange system, in response to receiving the secondary content opportunity information, the slot information, and sending, by the secondary content exchange system, to the content management system, the slot information.

6. The method of claim 5, wherein generating the slot information comprises determining the interest in inserting the secondary content based on the secondary content opportunity information.

7. The method of any one of the preceding claims, further comprising inserting the secondary content into the media stream in response to the user transmitting the feedback.

8. The method of claim 7, wherein inserting the secondary content comprises receiving, by the client device, the feedback from the user; and sending, by the client device, to a third processing system, hereinafter referred to as "secondary content stitcher", information, hereinafter referred to as "feedback information", indicating the feedback of the user.

9. The method of claim 8, wherein inserting the secondary content further comprises sending, by the secondary content stitcher, to the secondary content exchange system, information, hereinafter referred to as "forwarded feedback information", indicating the feedback of the user.

10. The method of claim 9, wherein inserting the secondary content further comprises determining, by the secondary content exchange system, based on the received forwarded feedback information, the secondary content to be inserted; and sending, by the secondary content exchange system, to the secondary content stitcher, information, hereinafter referred to as "secondary content reference information", indicating the secondary content to be inserted.

11. The method of claim 10, wherein inserting the secondary content further comprises receiving, by the secondary content stitcher, the secondary content reference information;HE Ref.: 269353 computing, by the secondary content stitcher, information, hereinafter referred to as "configuration file", based on which the client device can insert the secondary content into the media stream; and sending, by the secondary content stitcher, to the client device, the configuration file.

12. The method of claim 11, wherein inserting the secondary content further comprises inserting, by the client device, based on the configuration file, the secondary content into the media stream.

13. The method of any one of the preceding claims, further comprising, after receiving the slot information, setting, by the content management system, based on the slot information, the temporal reference at which the secondary content is to be inserted into the media stream.

14. A first processing system, hereinafter referred to as "content management system", for preparing content of a media stream, the content management system being configured for: sending to a second processing system, hereinafter referred to as "secondary content exchange system", information, hereinafter referred to as "secondary content opportunity information", wherein the secondary content opportunity information indicates an association between content, hereinafter referred to as "primary content", appearing in the media stream, and a temporal reference at which the primary content appears in the media stream; receiving, from the secondary content exchange system, in response to sending the secondary content opportunity information, information, hereinafter referred to as "slot information", indicating an interest in inserting content, hereinafter referred to as "secondary content", into the media stream; and inserting into the media stream, in response to receiving the slot information: information, hereinafter referred to as "announcement information", based on the received slot information and to be displayed at a client device, and an indicator, hereinafter referred to as "interactivity indicator", to be displayed at the client device, the interactivity indicator indicating to a user of the client device that the user, through the client device, can transmit feedback in relation to the announcement information.

15. The content management system of claim 14, wherein the content management system is further configured for: setting, based on the slot information, a temporal reference at which the secondary content is to be inserted into the media stream.

16. A system comprising:HE Ref.: 269353 first processing system, hereinafter referred to as "content management system", a second processing system, hereinafter referred to as "secondary content exchange system", a client device, and a third processing system, hereinafter referred to as "secondary content stitcher"; wherein the content management system is configured for: sending to the secondary content exchange system, information, hereinafter referred to as "secondary content opportunity information", wherein the secondary content opportunity information indicates an association between content, hereinafter referred to as "primary content", appearing in the media stream, and a temporal reference at which the primary content appears in the media stream; receiving, from the secondary content exchange system, in response to sending the secondary content opportunity information, information, hereinafter referred to as "slot information", indicating an interest in inserting content, hereinafter referred to as "secondary content", into the media stream; and inserting into the media stream in response to receiving the slot information: information, hereinafter referred to as "announcement information", based on the received slot information and to be displayed at a client device, and an indicator, hereinafter referred to as "interactivity indicator", to be displayed at the client device, the interactivity indicator indicating to a user of the client device that the user, through the client device, can transmit feedback in relation to the announcement information.

17. The system of claim 16, wherein: the content management system is configured for, before sending the secondary content opportunity information by the content management system to the secondary content exchange system: generating the secondary content opportunity information; the secondary content exchange system is configured for generating, in response to receiving the secondary content opportunity information, the slot information, and sending, to the content management system, the slot information; the client device is configured for receiving the feedback from the user, and sending to a third processing system, hereinafter referred to as "secondary content stitcher", information, hereinafter referred to as "feedback information", indicating the feedback of the user; the secondary content stitcher is configured for sending, to the secondary content exchange system, information, hereinafter referred to as "forwarded feedback information", indicating the feedback of the user; the secondary content exchange system is configured for determining, by the secondary content exchange system, based on the received forwarded feedback information, the secondary content to be inserted, and sending, to the secondary content stitcher, information, hereinafterHE Ref.: 269353 referred to as "secondary content reference information", indicating the secondary content to be inserted; and the secondary content stitcher is configured for receiving, by the secondary content stitcher, the secondary content reference information, and computing, by the secondary content stitcher, information, hereinafter referred to as "configuration file", based on which the client device can insert the secondary content into the media stream, and sending, by the secondary content stitcher, to the client device, the configuration file.

Images