User device, cloud backend, apparatus for performing a user application, related methods and computer programs using multiple detection information items

Abstract

A user device (100) comprises: a device hardware (102) comprising: a storage (102c) for detection information, wherein the storage (102c) comprises a plurality of storage entries (103a, 103b, 103c), each storage entry (103a, 103b, 103c) having a specific detection information item associated with a data instance identification of a specific data instance (401, 402, 403, 404, 405) of a plurality of different data instances (401, 402, 403, 404, 405), and a monitoring device (102a) for monitoring an environment of the user device (100); and a media runtime device software (110) comprising a media interface (114) to one or more data instances (401, 402, 403, 404, 405), and being configured for processing a monitoring result of the monitoring device (102a) to obtain a detected detection information item of a plurality of different detection information items, for accessing the storage (102c) to obtain the data instance identification of the specific data instance (401, 402, 403, 404, 405) associated with the detected detection information item, and for initiating a connection to the specific data instance (401, 402, 403, 404, 405) associated with the detected detection information item using the data instance identification of the specific data instance (401, 20 402, 403, 404, 405).

Description

[0001] User Device, Cloud Backend, Apparatus for Performing a User Application, Related Methods and Computer Programs using Multiple Detection Information Items

[0002] Specification

[0003] The present invention relates to data processing and, specifically, to the efficient and secure handling of data relying on a cloud backend support.

[0004] Typically, there exist user devices such as mobile phones, tablets, laptop computers or, generally, media rendering devices such as smart speakers or smart television sets. Such devices can access content providers such as Netflix, Amazon Prime, other streaming platforms or any Internet-based service for downloading data. Additionally, there exist internet platforms where users can upload media data. Such platforms are YouTube or Tiktok to only name a few.

[0005] Typically, a user has to manage, in her or his device, a plurality of application programing interfaces (APIs), which are required in order to interface with a provider platform in order to download or stream, for example, a movie. From the perspective of the user, this managing of different procedures in order to access different platforms is tedious, time consuming and increases the required hardware and software resources on the user’s device.

[0006] From the perspective of the content provider, the current situation is problematic in that there do exist many different smart devices for rendering the typically very valuable content. Some devices may be better protected against attacks in order to gain non-authorized access to a media stream than others. It is complicated for the content providers to contract with numerous hardware manufacturers in order to set minimum standards for data protection. However, this is necessary, since the most valuable good provided by the content providers is a protected content and, therefore, securing this protected content against non-authorized access is of ultimate importance for the content provider.

[0007] Typically, a modern user manages several applications by her or his mobile device. These applications, for example, refer to managing, for example, a first account at a supermarket, a second account at an electronic device shop and a third account at a certain media provider, for example, for downloading or streaming media data such as movies or films or for uploading data into the cloud. The managing of these different data instances that are typically remotely placed from the user represents a significant amount of managing operations to be done by the user.

[0008] Additionally, there typically exist user devices such as personal assistants, for example operated in mobile phones, which can be activated using so-called wake words or key words. A well-known wake word is, for example, “Alexa” in order to wake up or start a communication with the personal assistant from Amazon. Another personal assistant is “Si ri” used for communicating with an Apple device. A typical situation is that a device has a single wake word provided by the device producer such as Amazon for the example of Alexa or Apple for the example of Siri. These wake words are used by the devices in order to typically come back from a sleep mode and to start a communication, for example to connect to a cloud service or to operate smart phone devices, etc.

[0009] However, such wake words are bound to the devices, so that a user can wake up her or his device and then has to manually select a certain application such as the supermarket, the electronic device shop or the media provider.

[0010] It is an object of the present invention to provide an improved concept for handling different data instances.

[0011] This object is achieved by a user device in accordance with claim 1 , a cloud backend in accordance with claim 14, an apparatus for performing a user application in accordance with claim 28, a method of operating a user device in accordance with claim 34, a method of operating a cloud backend in accordance with claim 35, a method of performing a user application of claim 36, a computer program of claim 37, or a cloud system of claim 38.

[0012] A user device implementing the present invention comprises a device hardware including a storage for detection information such as a wake word or key word, wherein the storage comprises a plurality of storage entries, where each storage entry has a specific detection information item such as a wake word or key word associated with a data instance identification of a specific data instance of a plurality of different data instances. The device hardware comprises a monitoring device for monitoring an environment of the user device. Additionally, a software running on the user device as the media runtime device software comprises a media interface to one or more data instances and is configured for processing the monitoring result of the monitoring device to obtain a detected detection information item of a plurality of different detection information items. The storage is accessed using the detected detection information item to obtain the data instance identification of the specific data instance associated with this detected detection information item. In response to that, the media runtime device software initiates a connection to the specific data instance associated with the detected detection information item.

[0013] In contrast to earlier applications, a user is in the position to activate not just a device but actually a certain application running on the device using a detection information such as using a wake word or key word. To this end, the hardware device comprises a microphone as the monitoring device and this microphone continuously or triggered by a certain threshold monitors the environment of the user device. The trigger to start monitoring can also come through the HAL (e.g. if the device has a button to enable / disable listen). As soon as the user uses a detection information item such as a certain wake word, the device is not only activated from the sleep mode into the activated mode but the wake word results in a specific activation of a certain service or data instance and, in particular, together with the user information. Generally, this is only one of the possible options of reactions to the detection: One option is to resume from sleep mode; another option is to activate a certain service or a certain data instance, and another option is to resume from sleep mode and to activate a certain service or a certain data instance.

[0014] Since a user typically has several services configured on the user device, the user device is configured to hold several different wake words in a specific detection information storage, and the user device is in the position to not only detect a single wake word but to detect a plurality of wake words, i.e. , one for each wake word-enabled application. The service itself is not stored but enabled by a configuration, and the detection information is stored on the device.

[0015] This provides significant comfort for the user since the user does not have to care about which specific application is for which specific store, but the user can simply speak the corresponding wake word or show a picture of a corresponding video or image wake up information to a camera of the device, and the device starts to connect to this application or data instance already with the specific user ID or device ID (for which the cloud backend knows which user is the owner of the device) so that the user gets back her or his profile by just communicating the dedicated detection information item for the specific application.

[0016] Preferably, this service is provided in the context of a cloud backend support. From the perspective of the cloud backend, this entity comprises a detection information library for storing, for each one of a plurality of data instances, a detection information item associated with an identification of the corresponding data instance. The cloud backend has a user device interface for interfacing with a user device and a communication interface for interfacing with the plurality of data instances. Particularly, the cloud backend is configured to receive a message from the user device indicating that the user device has detected a call for a specific data instance stored in the detection information library and, additionally, the cloud backend provides a message to the user device for enabling the user device to access user specific metadata at the specific data instance of the plurality of data instances stored in the detection information library.

[0017] The cloud backend accepts the call from the user device indicating that the user device has been woken up and, additionally, this call already specifies not only that a wake word has been there but which specific wake word for which specific service was detected in the environment of the user device.

[0018] Hence, the cloud backend, by using the detection information library, is particularly suited for managing the further processing in order to establish a connection between the user and the corresponding data instance that has been identified by the specific detection information item. An additional advantage of this separation of tasks, i.e., that the user device does not directly interface with the data instance but via the mediation of the cloud backend is that the cloud backend can manage the connection to many different data instances even when these data instances require different protocols or application programming interfaces for being connected to.

[0019] The user is not required to always input any credentials for each specific data instance such as a web shop, but the user only has to provide the corresponding information of the different accounts for the different data instances or internet services a single time when onboarding these data instances. However, in the subsequent usage of this connection via the cloud backend, the user only has to authorize herself or himself with respect to the user device, but not anymore with respect to the different data instances, since this is automatically done via the cloud backend. In combination with a wake word call, and in connection with the cloud backend, a fully automatic and easy communication with the data instance is possible irrespective of whether the user has a small number of data instances or a large number. The only thing the user has to be aware is the certain wake word but, typically, this wake word is normally selected to be a close description of the service such as the name of a store, for example.

[0020] From the perspective of the user application or the apparatus for performing the user application or user app, this apparatus implements a user interface for receiving an input from a user such as a human user and, then, the user interface is a human-machine- interface. The apparatus additionally implements a cloud backend interface for communicating with a cloud backend and, in particular, the apparatus is configured to communicate, via the user interface 300, to receive user commands regarding a selection of a plurality of data instances for a communication. Additionally, a detected data instance message is received, via the cloud backend interface, indicating a specific data instance of a plurality of data instances and, additionally, a data instance confirmation message is sent out from the apparatus for performing the user application. In this embodiment, it is made sure that the user is prompted to confirm a certain wake word detection from the user device in the user application, typically, a user application software running in a mobile or stationary device, before the further procedure is performed. However, in other embodiments, this confirmation procedure is not required in order to establish the connection with the userspecific account information without further intermediate confirmation.

[0021] This rendering of the user specific account information such as a user start page is done on, for example, a display in the user device or is done in the display of the device on which the user application is running. Naturally, it can also be that the user application runs on the user device itself but this is not a requirement. In case of separate implementations of the user application on the one hand and the user device on the other hand, it is an embodiment that the user device has an interface for receiving a user input such as a microphone, a touch-sensitive display or any other human-machine interface in order to communicate with the data instance to the extent necessary, for example, to look up certain goods or services and to select a certain service for the purpose of buying or renting or leasing this service.

[0022] Subsequently, an implementation of an invention in a cloud eco system is discussed in more detail. A cloud system comprises several individual elements that cooperate with each other via corresponding interfaces. These interfaces can be hardware interfaces, logical interfaces, or mixtures between hardware interfaces and logical interfaces. The hardware interface of a user device can, for example, implement the control interface to a cloud backend and, at the same time, the media interface to a remote data instance. Contrary thereto, a user interface implemented by the apparatus for performing a user application will typically be a touch-sensitive display of the device implementing the user application in order to receive a user input while the cloud backend interface may be implemented by means of a hardware interface that may be a wired or wireless interface between the device, on which the user application is running, and an entry point of the cloud backend such as a hardware reachable via a cloud backend access address.

[0023] At the same time, a user device interface of the cloud backend and a communication interface for interfacing with a data instance such as a content provider or upload platform is implemented in a cloud backend together with the application interface for interfacing with a management application providing a user input. The realization of such interfaces can be done via hardware elements and / or software routines known for internet platforms or internet services and the related interfaces.

[0024] In an embodiment, a user device comprises a device hardware, a media runtime device software configured for controlling the device hardware, and typical means such as a processor and a memory for operating the user device. The media runtime device software comprises a control interface to a cloud backend device or platform and a media interface to a remote data instance. Particularly, the media runtime device software is configured for receiving commands for controlling the device hardware via the control interface and to receive or transmit content via the media interface.

[0025] The user device and, particularly, the media runtime device software which is implemented on the user device typically already from a device manufacturer makes sure that the full control of the device hardware is established via the control interface to the cloud backend and, on the other hand, the high volume data traffic to a typically remote data instance is done only via the media interface without introducing and such data into the cloud backend.

[0026] It is made sure that the high volume data traffic is not rooted within the cloud backend, since the media interface of the media runtime device software is not open to the cloud backend in the normal media streaming operation either for the purpose of downloading media or uploading media data.

[0027] A relatively smart procedure is possible in that the full control is done via the cloud backend without the high volume data traffic, while the high volume data traffic only takes place between the user device itself and the typically remote data instance selected for the purpose of uploading or downloading digital data such as media data.

[0028] On the other hand, a full control of what happens with the available media streams provided by the content provider typically against payment of a fee is made sure by means of the media runtime device software running on the user device which is configured for fully controlling the device hardware of the user device.

[0029] A further element of the cloud ecosystem is the cloud backend that comprises a user device interface for interfacing with a user device, a communication interface for interfacing with a data instance and an application interface for interfacing with a management application providing a user input. The cloud backend can be a distributed or concentrated system and, basically, comprises one or several processors and one or several memories. The cloud backend apparatus is configured to provide messages to the user device for controlling the user device to handle media data, and the cloud backend is additionally configured to receive messages from the user device via the user device interface. Particularly, the cloud backend is configured to communicate to the data instance via the communication interface only regarding an authorization of a user of the user device to receive data necessary for the authorization from the data instance or to send corresponding authorization data to the data instance. However, any media data are not exchanged between the cloud backend on the one hand and the data instances under consideration on the other hand. Only control data are exchanged so that the high volume data traffic that occurs, when a data stream from a content provider is established to a user device is kept out of the cloud backend in order to have smart and efficient cloud backend operation.

[0030] The cloud backend comprises an application interface for interfacing with a management application providing a user input. The user input is, for example, a request for a certain media stream or a request for uploading data captured by the user device, and such a request is forwarded from the user control application or user management application or user app to the cloud backend, and the backend receives this data via the application interface of the cloud backend.

[0031] A third constituent of the ecosystem is an apparatus for performing a user application or user app which once again comprises, as typically computer devices or smart devices, a memory and a processor. The apparatus for performing the user application comprises a user interface for receiving an input from a user and a cloud backend interface for communicating with a cloud backend. However, the apparatus for performing the user application does not have an interface to the individual user devices, since these individual user devices are only controlled via the cloud backend rather than by the user application directly.

[0032] The user interface is typically a human-machine interface such as a touch-sensitive display or a speech interface or any similar implementation. The cloud backend interface manages the communication to and from the cloud backend for the purpose of the selection of data instances such as data sources or content providers or data sinks, i.e. , upload platforms. A selection and control of the user devices is done via the cloud backend and a selection of media content for up or download is also done and managed via the cloud backend interface. Again, any high-volume traffic occurring when an actual stream is downloaded or uploaded is kept out of the user application apparatus, since this traffic only takes place between the user device itself and the data instance(s) via the media interface implemented by the media runtime device software running on the user device. Hence, when the ecosystem is considered as comprising the user device, the user application apparatus, the cloud backend and the (remote) data instances, each constituent has a very dedicated and limited interface situation.

[0033] The user application interfaces with the user via the user interface and with the cloud backend via the cloud backend interface. However, the user application does not interface with the remote data instances or with the user device via the media runtime device software configured for controlling the device hardware, when the user application is implemented on a device that does not also have such a media runtime device software. However, when the user application is also implemented in the same hardware device that also involves the constituents of the user device, then the media runtime device software nevertheless makes sure that the user application does not access the media interface or does not access the control interface to the cloud backend.

[0034] When looking to the cloud backend, the cloud backend only interfaces with the user application via the application interface, with the user device via the user device interface and with the remote data instances via a control interface for managing access and authorization issues. However, the cloud backend does not interface with the media interface of the user device or with the media port of the remote data instances by means of which the remote data instances output media streams or digital download data or receive media streams or digital input data.

[0035] On the other hand, the remote data instances do not interface with the user application but only interface, for control purposes, with the cloud backend and, for data receiving or data outputting purposes with the media interface of the user device. However, there is no possibility that the remote data instance communicates with the control interface operated in the user device.

[0036] From the point of view of the user device, it is to be noted that the user device is limited to only communicate with the cloud backend via the control interface implemented by the media runtime device software and to not interface with the user application and to only interface with the remote data instances via the media interface.

[0037] By means of these technical features, it is made sure that an efficient control is obtained that provides a high comfort for the user. This is due to the fact that the user only has to manage her or his account with the ecosystem via the user application, since the individual application programming interface (API) requirements for interfacing with different remote data instances are all handled in the cloud backend without further user interaction when the user has, typically at the first time of accessing the cloud service, installed the required login data at the remote data instances.

[0038] From the perspective of the data instances, the media runtime device software controlling the user device on the one hand and the cloud backend processing on the other hand make sure that the required data protection from the beginning to the very end point in the device hardware is respected and enforced. The fact that a translation of the different application programming interfaces required by the different data instances into a specific and generic protocol between the media runtime device software in the user device and the cloud backend is established, allow the operators of the remote data instances to implement changes easily. The changes only have to be adapted and respected by the cloud backend, but any changes at the data content provider do not result in any changes at the user device, and even do not result in any changes for the media runtime device software running on the user device, since the communication between the cloud backend and this service on the user device is always the same for each data instance and for each update of each data instance.

[0039] Preferred embodiments of the present invention are subsequently disclosed with respect to the accompanying drawings, in which:

[0040] Fig. 1 illustrates an implementation of a cloud ecosystem in accordance with an embodiment;

[0041] Fig. 2a illustrates an embodiment of the user device;

[0042] Fig. 2b illustrates an embodiment of the cloud backend;

[0043] Fig. 2c illustrates an implementation of a user application apparatus;

[0044] Fig. 2d illustrates an implementation of the ecosystem consisting of the user device, the user application, the cloud backend and the data instances;

[0045] Fig. 3 illustrates procedures to be done by a device maker for producing the user device;

[0046] Fig. 4 illustrates a communication sequence for the purpose of onboarding a device that acts as the apparatus for performing a user application; Fig. 5 illustrates a sequence of messages for the purpose of user information handover;

[0047] Fig. 6a illustrates a first portion of a sequence of messages between different entities for the purpose of playing content;

[0048] Fig. 6b illustrates a second portion of a sequence of messages for the purpose of playing a content and for the purpose of performing control playback during playing.

[0049] Fig. 7a illustrates an embodiment of the user device with a detection item storage;

[0050] Fig. 7b illustrates a specific embodiment of the storage of Fig. 7a;

[0051] Fig. 7c illustrates a sequence of procedures performed at least in part by the user device;

[0052] Fig. 8 illustrates a sequence of steps to be performed for updating a certain detection information in an embodiment; and

[0053] Fig. 9 illustrates an implementation of the cloud backend with a detection item storage.

[0054] Before discussion the figures in detail, an embodiment illustrating different aspects of the invention is illustrated.

[0055] A user device has a first account at a supermarket, a second account at an electronic devices shop and a third account at a media provider. Exemplarily, there can be a single account only, or only two accounts, three accounts or more than three accounts.

[0056] The supermarket, the electronic devices shop and the media provider all have different apps that are integrated in the Cinemo cloud service.

[0057] The user device (provided with the Cinemo device software such as the Cinemo SDK) contacts the Cinemo cloud service and notifies the cloud service of the three different accounts.

[0058] The Cinemo cloud service checks, whether it has an app for each one of the three suppliers. If an app is found (which is the case in this example), the Cinemo cloud service authorizes the user and the corresponding app with each other so that the user can trust the supplier and vice versa.

[0059] The supplier app in the Cinemo cloud service furthermore sends, from the cloud service to the Cinemo application running on the user device, a keyword or wake word for this supplier and this wake word is stored in the Cinemo application locally running on the user device.

[0060] Provided that each supplier has its own wake word, the result of this procedure is that the user device has stored three different wake words, one for each supplier

[0061] Now, the user device has stored several different keywords, such as the keyword for the supermarket, another keyword for the electronic devices shop and the further keyword for a media provider platform.

[0062] The user device continuously monitors its environment to detect a certain keyword of the plurality of keywords. As soon as a user device has detected the keyword e.g. for the supermarket, the user device contacts the Cinemo cloud backend.

[0063] The Cinemo cloud services check, whether the user is authorized (this is the case in this example). Hence, account information relating to the user’s account at the supermarket is available. The Cinemo cloud backend verifies the account information and establishes a connection between the user device and the supermarket. This establishing the connection can take place, in an example, in that the Cinemo cloud services send an URI (Uniform Source Identifier) or an URL (Uniform Resource Locator) on any other comparable information to the Cinemo application running on the user device.

[0064] Now, the user connects to her of his account at the supermarket website using this URI, URL or other related information and e.g. sends corresponding orders and receives corresponding replies from the supermarket website. Generally, there can be an exchange of audio or text or image information as the case may be.

[0065] Since the supermarket website detects that the URI, URL or other information come from the Cinemo cloud services and there is an application of the supermarket running on the cloud services, the user does not have to login at the supermarket each time the user connects to the supermarket. Only the first authorization is necessary and if this is successful, any further exchanges of credentials between the user and the supermarket website are not required.

[0066] The user device only communicates with the supermarket preferably without generating any traffic between the user and the cloud backend or the supermarket and the cloud backend. Instead, the Cinemo cloud services do not listen to this communication at all which is useful for the communication partners due to secrecy reasons and which saves bandwidth for communication since the only communication that requires more bandwidth takes place between the user and the supermarket website.

[0067] This embodiment is advantageous in that each supplier can establish its own keyword I wake word, and an update of any wake word is easily possible. If, for example, the supermarket intends to change its specific wake word, this only has to be communicated once to the Cinemo cloud services via the supermarket application located in the cloud services.

[0068] As soon as a user then connects to the cloud services to contact the supplier, the supplier application can initiate a wake word update at the Cinemo application locally running on the user device. This procedure takes place automatically without any further interaction between the supplier and the user device.

[0069] Fig. 7a illustrates a user device 100 comprising a device hardware 102. The device hardware comprise a storage 102c for detection information, wherein the storage 102c comprises a plurality of storage entries 103a, 103b, 103c, where each storage entry has a specific detection information item associated with a data instance identification of a specific data instance of a plurality of different data instances 401 , 402, 403, 404, 405. The storage 102c is illustrated, in more detail, in Fig. 7b showing the individual entries 103a, 103b, 103c. Each entry has a certain detection information for a specific data instance which is, for example, a store, and the corresponding identification is in the first column of the schematic illustration in Fig. 7b. Hence, the store or data instance with the identification ID2 has a certain detection information 2 which is a wake word 2, i.e. , for this store. In the example, the second data instance is an electronic devices store and, therefore, the wake word would be, for example, the name of this store while wake word 1 in entry 103a would be the wake word of the supermarket under consideration that operates as the data instance with ID1. In an embodiment, that there can be multiple wake words configured for a single service, so that a detection of not only one wake word but e.g. two wake words results in setting up a connection to one and the same data instance or store. The hardware comprises a monitoring device 102a which can be a microphone or a camera for acoustically detecting or visually detecting the detection information. In case of a visual detection scenario, the detection information would be a certain picture or a certain gesture made by a human. The device hardware comprises a processor 102b that is implemented to, in an embodiment, not only control the whole device hardware but to also perform a certain detection, but typically under the control of the media runtime device software 110. This software 110 comprises a media interface 114 to one or more data instances 401 to 405 and this media runtime device software is configured for processing a monitoring result of the monitoring device 102a to obtain a detected detection information item of a plurality of different detection information items. The runtime device software is configured for accessing the storage to obtain the data instance identification and to finally initiate a connection to the specific data instance identified by the certain detection information, i.e. , by the certain wake word using the data instance identification of the specific data instance.

[0070] Hence, for this purpose, the user device may only send the identification information of the certain data instance or the detected wake word or both information in order to notify the receiver of this message that a connection to the specific data instance is to be established. Hence, the sending out of such an information is a minimum implementation of the initiating a connection to the specific data instance. Further steps until this data connection if finally established can be done either by the user device or by a cloud backend service or by any other instance.

[0071] The media runtime device software 110 further comprises a control interface to a cloud backend. In case of a cloud backend service, the initiation of the connection to the specific data instance can be done by sending out a certain message that a certain wake word has been detected via this interface 112 but this can also be done in other embodiments via the interface 114 of Fig. 7a.

[0072] Subsequently, Fig. 7c is illustrated that shows a procedure mainly performed by the user device 100 of Fig. 7a. In a step 102, a monitoring of the environment using the monitoring device 102a such as the microphone or the camera is performed. This monitoring can be a continuous monitoring or only a monitoring at certain time periods or a monitoring that is activated by a certain trigger such as a detection of an increase of acoustic or optical energy around the user device. The trigger to start monitoring can also come through the HAL (e.g. if the device has a button to enable / disable listening). In step 804, the environment recording output by block 802 is processed in order to detect one of a plurality of predetermined detection information such as to find out whether a certain key word of a plurality of key words is included in the environment recording. Subsequent to the detection of a certain detection information item, step 806 performs a looking up of data instance information in the storage 102c in the user device in order to obtain the specific data instance that is associated with the certain wake word.

[0073] In step 808, the connection to the specific data instance is initiated. This initiating can be the triggering of a connection to the selected data instance either directly or via a cloud back end service or via any other means. As soon as the connection is established, the user device or a user application performs certain transactions to or from the selected data instance directly or indirectly as illustrated in step 810 in order to, for example, browse the content of a data instance and selecting a certain item from the content for the purpose of buying this item or for any other procedure that can be done via a user profile on a data instance. Another such application can be, for example, the entering of data in the web front end of, for example, an insurance company as a data instance or similar procedures. All these back and forth communications in order to perform such transactions are collectively included in step 810 of Fig. 7c.

[0074] The cloud-supported embodiment is specifically suited for updating operations that are, for example, necessary for data instances, when these data instances, for example, change their name. Naturally, changing the name of a store, for example, should also be migrated to all the individual user devices. However, this is all but easy since a supermarket provider, for example, does not have immediate access to all user devices that have an account at this data instance. A preferred procedure performed to a wake word change is illustrated in Fig. 8

[0075] When the name of a supermarket, for example, is changed, or when one supermarket takes over the other supermarket, the taking over supermarket is interested in also changing the wake word in order to make sure that the supermarket is called by its name rather than its (earlier) competitor’s name. To this end, the data instance in question, such as data instance 400, changes its specific detection information such as its wake word which is illustrated in step 820. This step 820 takes place within the data instance. In step 822, the data instance accesses the cloud back end and, particularly, this accessing takes place via the communication interface 204 of the cloud backend.

[0076] In step 824, a typical authorization step takes place so that the accessing data instance authorizes itself with respect to the cloud backend. In step 826, the cloud backend receives the new wake word from the data instance that has been correctly authorized and then stores the new detection information for this data instance and, preferably, also deletes the old detection information. This takes place in the cloud backend library 208 illustrated in Fig. 9.

[0077] In step 828, the cloud backend activates an update service for the specific data instance that has changed its wake word. This activation results in a generation of a certain information and / or message within the cloud backend that this specific data instance has changed the wake word. One result of this activation is that a login of a user device is specifically detected in step 830. Alternatively, the wake word change independent automatic detection of a login of a user device can also be specifically used by the cloud backend update service.

[0078] In step 832, the user device is asked for the data instance list, i.e. , for the individual data instances that have stored, within the user device, a certain wake word. Therefore, generally, the cloud backend asks the user device for basically the information in the first column of the storage content description of Fig. 7b. In step 834, the cloud backend receives this list subsequent to the user device having established this list and having sent this list to the cloud backend. Step 834 illustrates that the update service is initiated as soon as the specific data instance that has changed its wake word is located on the list provided by the user. The new wake word is communicated to the user device and, in step 836 performed by the user device, the old detection information in the storage is replaced by the new detection information in the specific storage place for this specific data instance.

[0079] Fig. 2a illustrates a user device 100 comprising a device hardware 102 and a media runtime device software 110. This media runtime device software is running on the user device and is configured for controlling the device hardware as illustrated by the connection between block 110 and 102. The media runtime device software comprises a control interface 112 to a cloud backend in order to exchange messages between the media runtime device software in the device and the cloud backend. The user device comprises a media interface to a typically remote data instance for connecting to a remote data instance typically via a playable URL (unique resource locator) in order to then obtain a stream of media data from the remote data instance. Alternatively, the media interface can also send a request for a URL to the remote data instance and, then, receives this data from the data instance via the media interface and, then, an upload from the user device 100 to the remote data instance can be performed when the data instance is a data sink such as the YouTube platform or the TikTok platform. The data that are uploaded via the media interface 114 are typically data acquired by the user device hardware, which is, for example, a camera or a microphone.

[0080] For replaying a data stream received via the media interface 114, the user device typically has a display or speakers or both. Particularly, the media runtime device software 110 is configured to receive data for controlling the device hardware via the control interface 112 and to receive or transmit content via the media interface 114. It is made sure that the high volume data traffic only takes place via the media interface 114, and this high-volume traffic is kept out from the cloud backend, since the media interface 114 is not interfacing with the cloud backend.

[0081] In an embodiment, the media runtime device software is configured to receive, via the control interface, metadata comprising information identifying the data instance under consideration and an access information such as a playable URL combining the metadata and the identification of the data instance to be accessed. Based on this information, the media runtime device software is configured to access the data instance using the received access information via the media interface 114.

[0082] In one embodiment, the data instance is a content provider. In such an implementation, the media runtime device software 110 is configured to receive playback metadata identifying an intended content data for playback and the access information via the control interface 112, and to send the playback metadata to the content provider via the media interface 114 and to receive the content for playback from the content provider via the media interface 114.

[0083] In the embodiment, the user device comprises a hardware abstraction (HAL) layer 11 interfacing the device hardware on the one hand and the media runtime device software 110 on the other hand. The hardware abstraction layer 11 is configured for detecting a playback event. The media runtime device software is configured to receive the playback event from the hardware abstraction layer 11 via control line 26. However, this playback event is not immediately executed, but is forwarded to the cloud backend via the control interface 112. When a confirmation on the playback event from the cloud backend is received via the control interface 112, then the hardware abstraction layer 11 is instructed by the media runtime device software 110 to perform the playback event. However, when no such confirmation for the playback event is received from the cloud backend or when an explicit “action forbidden” message is received from the cloud backend, then the playback event is not executed. It is made sure that the content provider has full control not only on the fact that an unintended decryption is forbidden, but also how the media data is handled in the user device.

[0084] Nevertheless, in order to give the user of the user device as much control as possible, a pressing of a certain switch on the user device which is typically, a start button, a stop button, a fast forward or fast rewind or pausing button, is nevertheless detected, but not immediately executed as is done, in contrast, in prior art devices. Instead, this hardware event is forwarded to the cloud backend and only when the cloud backend confirms this event, then this event is actually executed.

[0085] The user device comprises a secure storage for storing a unique device ID which is required for identifying the user device with respect to the cloud backend. The device hardware is also configured to only replay content identified via a command received via the control interface and received via the media interface. In a further embodiment, the media runtime device software or “Cinemo SDK kit” is configured to also have a web browser functionality 21 , a media player functionality 22, and a content decryption functionality 28. In a further embodiment, the media runtime device software 110 is configured to receive a playable unique resource locator (URL) from the control interface, and the user device then interfaces with the specific remote data instance, from which the playable URL is coming from via the media interface to stream media data located at the playable URL at the content provider via the media interface 114. Other ways to access a data source or data sink platform different from via a playable URL can be performed as well.

[0086] Fig. 2b or Fig. 9 illustrates an implementation of the cloud backend 200 that has a user device interface 202 for interfacing with the user device such as the user device 100 of Fig. 2a. The cloud backend comprises a communication interface for interfacing with a typically remote data instance such as any one of the data instances indicated at 400 in Fig. 2d, where individual data instances 401 , 402, 403, 404, 405 are illustrated. These remote data instances also correspond to the media CDN 5 of Fig. 1.

[0087] The cloud backend is configured to provide messages to the user device interface for controlling the user device to handle media data and to receiving messages from the user device via the user device interface 202. The cloud backend is configured to communicate to the data instance via the communication interface 204 regarding an authorization of a user of the user device, where this communication comprises receiving data from the data instance or sending data to the data instance. However, this data does not comprise media data to be downloaded or to be uploaded from the user device to the data instances. The cloud backend comprises an application interface 206 for interfacing with a management application such as the user application 300 illustrated in Fig. 2c or 2d.

[0088] The cloud backend 200 additionally comprises, for performing the corresponding tasks, libraries / data bases 208 that can be distributed in the cloud or that can also be located on a single server. The cloud backend also comprises a processing engine 210 that can also be a distributed processing engine or a concentrated processing engine. The playback / upload control of the user device is performed by the cloud backend typically without high-volume traffic in the cloud backend. Only control data is communicated within the cloud backend and control data is received via the cloud backend interfaces 202, 204, 206 and corresponding control data is output via these interfaces as well.

[0089] Fig. 9 illustrates an embodiment, where the databases 208 in Fig 2b comprise a wake word database 208a, in which for each data instance exemplary illustrated as a store, a store ID is maintained in relation to a wake word WWn for this store with number n. The information may additionally comprise other information such as a name of the store, an information on the store e.g., how the store can be contacted or which authentication protocol is required by the store such as the Oauth2 protocol or any other necessary or useful information on the data instance I store.

[0090] Fig. 2c illustrates an apparatus 300 for performing a user application which comprises a user interface 302 which is exemplarily a human machine interface such as a touch screen interface or a speech interface in order to provide messages to a user or to receive messages from a user. The apparatus comprises a cloud backend interface 304 for communicating with a cloud backend such as the cloud backend of Fig. 2b or Fig. 9 or the corresponding elements located in the cloud backend in Fig. 1 comprising a playback management element, device queues, or a device management element 35 for example. The apparatus 300 can, in a specific embodiment, also include a user device interface 303 for directly interfacing the media device runtime software 110 running on the user device 100 for performing a direct communication between the user device 100 and the user application without the cloud backend in between. This direct connection can be limited to an operation of the user specific interface at the data instance under the prerequisite that the user specific interface has been requested and initiated via the cloud backend 200.

[0091] The apparatus is configured to communicate, via the user interface 302, to receive user commands regarding a selection of one or more data instances and a selection of content to be sent to a data instance or to be retrieved from a data instance and to display messages for the user, to communicate via the cloud backend interface 304 regarding the selection of the one or more viewed data instances to be accessed by the user devices, and to communicate via the cloud backend interface regarding the selected content for sending from the user device to a corresponding data instance or for sending from the data instance to the user device. Hence, the cloud backend interface 304 cares for the selection of data instances for the selection and control of user devices, and for the selection of media content for the upload and download.

[0092] Fig. 2d illustrate an overview of the cloud ecosystem and how the individual interfaces are connected to each other in the four greater “entities”:

[0093] The user device 100 with the media runtime device software 110. The user application apparatus 300 with the user interface 302 on the one hand and the cloud backend interface on the other hand.

[0094] The remote data instances 401 , 402, 403, 404, 405, with the high volume data connection 51 between the media interface 114 of the user device 100 and the media interface of the corresponding data instances.

[0095] The cloud backend for communicating with the remote data instances, the user device 100 and the user application apparatus 300.

[0096] The user application apparatus 300 for communicating with the cloud backend and the user herself of himself.

[0097] In an embodiment, the user device is responsible for playback of content from the remote data instances, and this device, in a minimum implementation, only consists of logic to ensure this capability and, additionally, has the ability to display the playback state in case of a screen connected to the user device or integrated in the user device. The user device only reacts and performs actions based on commands received from the cloud backend via the control interface 112 and there is no local ability to change what is being played. The user device only operates in conjunction with the cloud backend except for already cached or pre-fetched content which will continue to play even without the connection to the cloud backend. Any buttons included in the hardware of the user device can remain active, but not active in the sense that a command implemented by a button is immediately executed, but only active in the sense that the command is detected, forwarded to the cloud backend, confirmed by the cloud backend and only when the confirmation is there, executed in the hardware of the user device. The user device supports all playback capabilities provided by the media runtime device software including playback of DRM protected content and synchronized playback.

[0098] Devices on the same local network can be used in a group playback leveraging synchronized playback, where the synchronization and distribution of content is performed locally. This is realized by electing one device as the primary device, this primary device will receive a content stream and will distribute it over the local network to all other participating devices. This primary device is elected algorithmically among all participating devices and can change in the course of active playback, specifically in case of devices becoming online or offline in the course of playback.

[0099] The user device also provides an interface for onboarding purposes that allows an app on a mobile device to provide network information to the user device in order to connect it to a local network. The actual hardware used in the user device is, in an abstract representation, a hardware abstraction layer being provided by a device maker and used together with the media runtime device software. The HAL is used to control network interfaces, receive signal input signals and receive status information for further use from the media runtime device software 110, among other needed abstractions to interact with provided hardware such as audio or video output. The device always identifies itself towards all other participants in the ecosystem based on a unique device ID securely stored on the user device 100.

[0100] The cloud backend comprises multiple services with detailed responsibilities. The cloud backend 200 is the main control and management interface within the ecosystem, where all changes in the ecosystem are managed and controlled via the cloud backend 200. Exemplarily, the cloud backend comprises a device management element 35 illustrated in Fig. 1. In this device management, it is made sure that every user device is registered as owned by a user. Users have the ability to see their own devices, to name these devices, to tag these devices, and to deregister them. Users can create and delete device groups which allows using a device group like a single device.

[0101] The cloud backend comprises a playback management that enables the users of the ecosystem to control various playback aspects of the owned devices of the user. Exemplarily, playback of audio, images, and videos is supported as well as an upload of audio data, video data or image data or media data.

[0102] Users can control single devices, view content from content providers and control the playback thereof independently from the local network. This means that the controlling device such as a mobile phone, browser or tablet or generally the user application apparatus 300 can be connected to any network. The playback control includes start, stop, seek, skip, and other functionalities offered by the content provider the content is coming from. Restrictions set by the content provider are respected, for example, skip limits, etc. Users can create playlists that can be composed of items from any content provider of any supporting media type. The user-created playlists can be queued on a device owned by the user. The device can offer any playback control operations directly via buttons and the user can control playback directly from there. However, such control via the buttons is not immediately executed but is only executed when confirmed by a message originating from the user device interface 202 of the cloud backend 200. Users can control playback targeting a device group containing arbitrary devices the user owns. Each separate network segment of the device group will play the content synchronously and technically act as one device. Users can control playback targeting a virtual device and the content will be played according to the virtual device definition.

[0103] Additionally, the cloud backend comprises a content service management that allows the end user to connect to content services or, as named before, data instances that are integrated via the functionalities performed by the user application apparatus 300. End users need to have a subscription / account for the corresponding service at the remote data instance they want to connect and the corresponding user application must allow the user to have access to the service.

[0104] The cloud backend additionally comprises a media app management functionality that defines all meta data for content services, decryptions, translations, icons as well as a specification on how to access content via an API or an URL or URI for web applications. The media app management also allows service providers to specify the requirements their service has towards a device (codecs, screen resolution, DRM, etc.). It also allows a content service to configure restrictions for the service allowing to restrict a service availability by device meta data (manufacturer, type, model, etc.), regions or accounts. These restrictions can be formed by using deny lists and allow lists.

[0105] Additionally, an update management is provided in the cloud backend. In general, devices receive updates of their running device media player software over the air. For that, the devices check frequently if there is a new software version available for a specific device model and install it accordingly. The update will be installed when the device is not performing playback, however, devices performing continuous playback will forcefully install the updates after a defined time. Update packages are stored on the distributed CDN (Content Delivery Network) illustrated at 5 in Fig. 1 to ensure high availability and low latency. An update package contains a specific version of the media runtime device software that runs on the device and informative meta data for the update mechanism to be able to select the correct update package. Update packages can be private, so that they can be restricted for development purposes for specific manufacturers, device models or users. The updates are managed through the cloud backend 200 and the device manufacturers. Manufacturers can define if their device models receive a specific version of the update or not.

[0106] The user application apparatus 300 relies on rendering the web frontend exemplarily within a native application. On top of rendering the web frontend, the user application is also integrated in a playback device also allowing remote control via the web frontend or other applications. On top of these functionalities, a further purpose of the user application apparatus 300 is to provide support for onboarding of devices.

[0107] The onboarding of devices is happening by using a camera of a mobile device to scan a QR code. The QR code contains information about the device ID (uniquely identifying the device) as well as optionally information about a local wireless access-point of the device. After scanning the QR code, the user application 300 will connect the mobile device to the wireless access-point of the device. In the next step, the device itself is providing the mobile app with a list of available wireless networks that the device can connect to. The user can select the wireless network to connect to and will be asked to provide the necessary wireless network password. This information is transferred from the user application to the device on which the user application is running and, then, a connection to the wireless network and a connection to the cloud backend 200 are established.

[0108] Subsequently, further aspects of the present invention are discussed in more detail. A further element that is mentioned in Fig. 3 is a device descriptor. Particularly, as illustrated in Fig. 3, a device maker 500 is performing several procedures to generate a cloud serviceready user device. Particularly, the device maker produces a hardware device and, then, as illustrated in item 501 , puts the media runtime device software on the device. As illustrated in item 502 a set of unique device IDs is acquired and, then, as illustrated in step 503, the device IDs are securely put on the devices, one device ID per device. The device manufacturers create and configure a device descriptor as illustrated in item 504 and, additionally, set rules for allowing or disallowing certain media apps as illustrated in 505. In an embodiment, these rules are stored in the cloud backend and, particularly, in the certain libraries or databases of the cloud backend illustrated at 208 at Fig. 2b or 208a at Fig. 9. Fig. 4 illustrates an example procedure of onboarding of the user application apparatus 300 to become a user device. This is done implicitly after the user 99 downloads 601 the app from a store and logs into the app for the first time. Particularly, for this purpose, the user 99 starts the app in step 602, and the user application receives, via the user interface 302, a starting message of the app 602. The application apparatus 300 displays a login screen and receives a login message 603 from the user 99.

[0109] When the user is logged in, the application 300 receives a device ID that is internally assigned to the user so that the user becomes the owner of the user device, on which the user application is running, i.e., the owner of the user application apparatus. In an embodiment illustrated in Fig. 4, the user application sends out credentials to the cloud backend 200. This occurs by sending out the message 604 via the cloud backend interface 304 of the user application to the application interface 206 of the cloud backend. In reply to this message 604, the cloud backend 200 replies, via the application interface 206, with a message 605 comprising a token, which is received from the user application 300.

[0110] Using this token, and when the starting is performed for the first time as illustrated in item 606, the application retrieves, from the apparatus, on which the user application is running, a MAC information (media access control information), or an IM El (International mobile equipment identity) information and sends this data via a message 607 to the cloud backend 200 via the application interface 206 of the cloud backend. In response to this data, the cloud backend creates a device ID as illustrated in block 608 and, additionally, stores device data derived using the IMEI or the MAC data and outputs the device ID using a message 610 via the application interface to the cloud backend interface 304 of the user application. The user application then stores this device ID in the apparatus, on which the user application is running. Now, the device, in which the user application 300 is running has a unique device ID known and managed by the cloud backend 200. Therefore, in order to be able to finally become a device, it is only required that, for example, using the device ID, the media runtime device software already existing on the apparatus, in which the user application is running, is activated, or this data is downloaded so that the device, in which the user application is running, is implemented to be a compliant device.

[0111] Fig. 5 illustrates a further sequence of messages between the user 99, the user application 300, the cloud backend 200 and the data instance or content service 400 for the purpose of performing a user information handover. A user has access to all content services or data instances that are allowed to be used by the user. To be able to use a specific content service within and under the control of the cloud backend, the user has to authorize the specific content service or data instance. For this purpose, the user 99 sends an authorized content service message 620 to the user application via the user interface 302 and the user application receives this message 620 via its user interface. In response to this message 620, the user application requests an authorization of the content service for this user via the cloud backend interface 304 and a message 621. In response to this message 621 received by the cloud backend 200 via the application interface 206, the cloud backend looks-up a certain authorization method which is offered by the content service under consideration using a certain database where this information is stored in the cloud backend. When this specific authorization method is found by the cloud backend, the cloud backend sends an authorization request message 622 to the content service or remote data instance via the communication interface 204 illustrated in Fig. 2 in order to access the remote data instance under consideration, but not for media download or upload but only for control purposes. In response to this request 602, the content service replies with a request 623 to the cloud backend 200 which is forwarded from the cloud backend 200 to the user application 300. In the message 223, the user application is required to provide credentials i.e., login data for the specific content service 400 and exemplarily also to provide the consent to the terms required by the specific content service. This data is input from the user 99 via the user interface (not illustrated in Fig. 5) and sent back via message 624 to the cloud backend and from there to the content service 400. In response to that, the content service sends an access token via message 625 and additionally, also a refreshment of the access token as required under the specific circumstances is performed. The cloud backend stores this token and typically also the expiration of the token, as illustrated in 626, for further use in a playing content functionality illustrated subsequently with respect to Fig. 6a or 6b. In a step 628, the detection information such as the wake word for the specific data instance is sent to the user device and is stored 629 in the specific storage 102c of the user device exemplary illustrated in Fig. 7a.

[0112] After successful authorization, the ecosystem stores the tokens together with the user ID so that they can be fetched as soon as the content service API is accessed. The tokens are kept only in the cloud backend and do not have to be managed by the application 300 in any sense. Due to the fact that the user has authorized the content service a single time, a next access to the content service via the cloud backend by the user application will not require any further input of the credentials or any passwords by the user. When a list of authorized content services is needed, a database included in block 208 of the cloud backend 200 (Fig.2b) is queried for all content services where tokens for the user under consideration exist.

[0113] Fig. 6a and 6b illustrate a procedure showing a sequence of steps / message between the user 99, the user application 300, the cloud backend 200, the media runtime device software 110 and the content service 400 for playing content with the cloud ecosystem. The playing of content or the uploading of content within the ecosystems involves actions of the corresponding participants of the ecosystem such as device makers, content services, users. It is to be noted that the heavy-weight data or content is retrieved directly by the device media runtime software of the user device, while the cloud backend only handles the playback commands. The data of authorized devices and authorized content services for a user are stored and retrieved from databases that are part of the cloud backend. These databases contain all data that is necessary for authorized access to a content service, where tokens are needed for access. When, in the Fig. 6a, Fig. 6b embodiment such data is looked-up in the cloud backend, it is retrieved from a corresponding database 208.

[0114] It is to be noted that the illustration in Fig. 6a makes clear that each message, such as a message 700 between the user application 300 and the cloud backend 200 refers to two actions of the entities involved. The first action is that the user application sends out the message 700 via the cloud backend interface 304, and the second action performed by the cloud backend 200 is that the cloud backend 200 receives the message 700 via the application interface 206. For each of the corresponding messages that are exchanged between two entities in the Fig. 6a, Fig. 6b illustration or in the Fig. 5 and Fig. 4 illustrations, there exist the two actions, i.e., on behalf of the entity sending the message and on behalf of the entity receiving the message, where the corresponding interfaces involved are clear from the illustration. Hence, although not explicitly outlined all the time, these two actions are always performed.

[0115] Subsequently, Fig. 6a and Fig. 6b are discussed in more detail. In the action 602, the user 99 opens the user application running on the user application apparatus. In step 603, the user logs into the app and, in reply to this logging into the app, the user authenticates herself or himself via the message 700 from the user application to the cloud backend. In order to further use the cloud backend, the user receives a token including the user’s permissions and the corresponding user ID in message 605. The user token is used in all calls to the cloud backend to identify the user and to check if access is allowed for the requested function for the user.

[0116] In order to finally come to a playing of a media content, the user application sends a request for a list of content services illustrated at 702. All content services that are authorized in the cloud backend are returned. Each content service is either authorized or not authorized by the user. Hence, due to the fact that the cloud backend replies to this message 702, with a full selection of content services authorized in the ecosystem, the selection what is valid for the specific user is performed in step 704. Here, the cloud backend checks, which content services are actually authorized for the user.

[0117] For each identified content service, it is checked, if the user is allowed to use the content service. If the user is not allowed, the content service is removed from the result. In a step 708, it is checked, whether the content service allows the user. Probably, the user is denied for some reason, for example by not having paid a bill, etc. This is done in step 708. In step 710, the cloud backend returns a list of allowed content services to the user application 300. This list is presented to the user via the user interface in the user application.

[0118] In step 717a, the user device and specifically the media device runtime software 110 detects a wake word of a plurality of stored wake words. In step 717b, a look up to the storage in the user device is performed, and the data instance identification information is retrieved and used for forming a message in step 717c.

[0119] The user device sends, subsequent to the detection of e.g., a wake word, the message 715 to the cloud backend for activating a specific user instance which can be a store, such as a supermarket, a shop for electronic devices or a supplier of media data, such as streaming data or any other material or non-material goods or services. In response to that, the cloud backend looks up, whether credentials for this content services are stored in the cloud backend. This is done in step 716, and when such user credentials for the requested content service are found, an authentication message 718 is sent from the cloud backend to the content service via the communication interface 204 of the cloud backend. The result message 720 is illustrated in Fig. 6a and the result message can, for example, comprise a certain token, from the content service, or any other positive authentication result.

[0120] In response to the positive result 720, the cloud backend requests content metadata in a message 722 from the content service 400, and the content service 400 sends back the requested content metadata in message 724. This content metadata includes an information in the detected data instance belonging to the detected wake word. In the embodiment illustrated in Fig. 6a, a message 725 comprising this data instance information is sent to the application apparatus 300 and displayed to the user 99 via the user interface. The application apparatus 300 receives a user confirmation message 746 via the user interface 302 of Fig. 2c. In response to a positive confirmation message 746, a connection activation message 748 is sent to the cloud backend 200. In order to be able to access the data instance under consideration, content metadata such as a playable URL or URI is sent from the cloud backend 200 to the media runtime device software 110 running on the user device 100. Hence, the content metadata are sent from the cloud backend 200 to the media runtime device software 110 on the user device. As stated, this content metadata can, for example, be the playable URL or any other data, by which the media runtime device software 110 running on the user device can request the content via a message 752 from the user device media interface 114 via the connection to the typically remote data instances illustrated at 51 in Fig. 2d. As soon as the playable URL, for example, is received by the remote data instance, the remote data instance streams back the content as shown in message 756 and this takes place via the media interface 114 of the user device 100 and the content is started to be replayed as illustrated at 754. For replaying the content, the content is downloaded 756 from the data instance. It is a specific feature of the embodiment, that the content received in step 756 is already a content specific for the user at the specific data instance selected by the wake word detected in step 717. Such a content specific for the user can be the products bought by the user in earlier sessions of suggestions for the specific user made by the data instance or any other user specific data instance interface personalized by the user under consideration. The user ID is forwarded by the cloud backend to the content service in the message 752, but can also be forwarded earlier or at any other stage in the processing, i.e. , in message 718 or 722.

[0121] In step 756, the user-specific content such as the user- personalized data instance interface is rendered by the user device. This rendering can take place on the user device itself via a specific interface 770 such as a touch sensitive interface on the user device. The (human) user can input a selection of a certain item via the interface 770 and this selection is forwarded to the data instance in step 772, and the data instance can react in step 774. Generally, a dialogue can take place between the user 99 and the data instance 400 via the interface 770.

[0122] As illustrated in Fig. 6b, other alternatives for this communication can be implemented as well. One alternative is that the content received in step 756 is forwarded to the cloud backend 200 in step 758 and from there, via message 760, to the user application running on device 300 provided that the device 300 is (physically or logically) separated from the user device. The user reaction 762 such as a selection of a specific item from the user- personalized interface from the data instance 400 is received and the selection is forwarded 764 to the cloud backend 200 and from there via message 766 to the media runtime device software 110. This instance sends the user-selection retrieved from message 766 to the data instance in step 772 using message protocol information such as a playable URL received from the cloud backend 200 in message 766, and a corresponding reply is received in step 774. Other communication protocols apart from using playable URLs can be implemented as the case may be. A further alternative is the direct communication between the media runtime device software 110 and the user application 300 as illustrated by the item 776 illustrating a back and forth communication between items 300 and 110. To this end, the user application device 300 implements a user device interface 303.

[0123] Generally, the user device 100 can communicate with the remote data instance via several different ways wherein each one of the plurality of data instances 401 , 402, 403, 404, 405 comprises a user-specific communication interface for communicating with the user device. The media runtime device software 110 is configured, in the initiating a connection to the specific data instance 401 , 402, 403, 404, 405 associated with the detected detection information item, to send 752 a request for a user-specific content to the specific data instance, to play 754 the user-specific content; to receive 768, 766, 776 a user reaction to the user-specific content; and to communicate 772, 774 with the user-specific communication interface in response to the user reaction.

[0124] It has to be noted that the “playing” in step 754 and 756 does not necessarily imply that the information from the data instance is actually displayed or rendered audible by the user device. In case of an involvement of the user application, the “playing” in steps 754 and 756 is a forwarding of this content or an information how this content can be made visible of audible to the cloud backend and from there to the user application 300 or without involvement by the cloud backend to the application 300 directly.

[0125] The media runtime device software 110 is configured, in receiving 768, 766, 776 a user reaction to the user-specific content, to receive 768 a user input via a user interface 102d included in the device hardware 102, or to send 758 information on the user-specific content to the cloud backend 200, and to receive 766 a user reaction to the user-specific content, or to directly communicate 776 with an apparatus 300 for performing a user application to obtain a user reaction to the user-specific content

[0126] Subsequently, features of embodiments are summarized that make up a preferred cloud ecosystem. Other embodiments of the cloud ecosystem can include more or less parts or roles than described below. An embodiment consists of multiple parts and roles as described subsequently:

[0127] 1 . Cinemo - the inventor and operator of the new Cloud Ecosystem

[0128] 2. Cloud Ecosystem - a uniform cloud media platform connecting Users, Devices, and Content Services via the Cloud Backend and providing the Management Services. 3. Device Maker - any party designing and / or manufacturing Devices. It includes also private technical savvy users acting as such Device Maker by installing the Media Runtime Device Software on a device of their own (e.g., a private camera connected to an SoC of a User).

[0129] 4. Device Brand - a commercial brand under which a Device Maker offers Devices

[0130] 5. Content Services - any party having cloud services for sending (downstream from cloud service) or receiving (upstream to cloud service) content on or from a Device (examples for sending are e.g., services like Spotify, Netflix, or advertising content services, business radio services, private streaming services, content management systems (including systems handling scheduling, delivery and curation) and for receiving can be e.g., a service to store digital camera data in the cloud).

[0131] 6. Media App - a descriptor defining and enabling the connection between the Content Services, the Cloud Backend and the Device. And enabling a Content Service additional functionality by communicating with the Device via the Media Runtime Device Software, like e.g., like saving content on a Device. A Media App is only available in the Cloud Backend, the Device and the Media Runtime Device Software do not have information about the Media App, providing additional security and separation of concerns.

[0132] 7. Device - a hardware device having at least one CPU and a connection to the internet, having Device Capabilities and having integrated the Media Runtime Device Software. A Device is uniquely identified by a Device ID and can have a User as its owner.

[0133] 8. Media Runtime Device Software - embedded software running on a device and communicating both with the device hardware and with the Cloud Backend, and supporting Formats and other capabilities depending on the Device Capabilities. The Media Runtime Device Software is independent of specific Media Apps, and is compiled to many OS and SoCs systems.

[0134] 9. Cloud Backend - cloud services enabling and maintaining the connection and control of the Media Runtime Device Software of all Devices. Through this connection enabling the Cloud Ecosystem. The Cloud Backend has its own Web APIs (e.g., REST APIs). Media Apps are used in the Cloud Backend to communicate content for playback from and to Devices.

[0135] 10. Device Capabilities - the list of all hardware capabilities of a device supported by the Cloud Ecosystem. This includes all supported Formats, and including capabilities like supported audio and video codecs for encoding and for decoding, performance capabilities, security capabilities such as secure boot, DRM capabilities, payment capabilities, etc.

[0136] 11. Device ID - a unique identifier for each Device 12. App - an application by Cinemo using the Management Services to manage the Devices and the Account. Optionally, an App can have its own Media Runtime Device Software making the App a Device itself. An App can be a mobile app running on a smart phone or tablet or can be running in a browser or an application running on a PC.

[0137] 13. User - a user with an Account. Users can own their own Devices or use Devices of any other user (e.g., colleagues, or family members).

[0138] 14. Role - each User has one or more roles to define the rights of the User

[0139] 15. Account - a unique user with a free or paid subscription to the Cloud Ecosystem

[0140] 16. Formats - at least one of audio, video, pictures, text and interactive (web) content playing / displaying, and / or generating at least one of audio, video, pictures, text and interactive (web) content. Examples for playing are (i.) speakers for audio, (ii.) screens, TVs for video and pictures, (iii.) screens, keyboard for interactive content. Examples for generating are (i.) microphones for audio, (ii.) cameras for video and pictures.

[0141] 17. Play, Playback, playback or played or play - is broadly defined as playing Formats on a Device and / or generating Formats on a Device

[0142] 18. Store - a service of the Cloud Backend enabling Users to search, find, select and unselect Media Apps and authenticate them with the new Cloud Ecosystem using the Content Services credentials, and vice-versa. This allows the Cloud Ecosystem to e g., access content from Content Services.

[0143] 19. Management Services - Services in the Cloud Ecosystem available via the Cloud Backend for participants to control and manage some or all aspects for one or many Devices (e.g., managing Devices including onboarding and offboarding, play queue management, playback management, etc.).

[0144] 20. Device Descriptor defining and enabling the connection between the device maker, the Cloud Backend and the Device. The Device Descriptor contains at least the Device Capabilities, a device name, a device picture and the device type.

[0145] An embodiment is a system consisting comprising:

[0146] - An unlimited number of Devices

[0147] - An unlimited number of Users

[0148] - An unlimited number of Media Apps

[0149] Enabling access to each User’s Devices and Content Services over Apps and / or Management Services, with the possibility to delegate rights to other Users.

[0150] Giving Content Services full control over (i.) minimum Device Capabilities a Device must have to be allowed and able to play their content (for example codec support, secure boot, DRM, etc.) for the purpose of e.g., ensuring best possible playback quality, and / or the minimum security requirements (ii.) Devices from which Device Brand(s) will be allowed to play their content, or to not play their content, (iii.) which Devices get access to which Media App enabling a Content Service for example to exclusively one Device Brand, or any combination / permutation, (iv.) allow Content Services to extend their own applications the capability to manage Devices with their own Media Apps (e.g., remote control).

[0151] Giving Device Makers full control over (i.) which Media Apps they want to allow on their Devices, (ii.) which Device Capabilities of the Cloud Ecosystem the Device should support and make available to the Cloud Ecosystem, (iii.) allow Device Makers to extend their own applications the capability to manage Devices via Management Services (e.g., Device onboarding to the Cloud Ecosystem).

[0152] Giving Users full control over (i.) selecting Media Apps via the Store and authenticate the Cloud Ecosystem with them, (ii.) using Apps and the Management Services to manage and control the Devices they own, (iii.) permissions of the Media Apps which Device Capabilities they can access via the Management Services (e.g., fora Device with a speaker and a microphone, a User should explicitly allow the Media App to control playback on the speaker, and to play from the microphone, i.e. to access the microphone input and send it to the Content Service), (iv.) delegating User’s rights partially or fully to other Users using e.g., Roles.

[0153] Advantages of the embodiment are, amongst others: Separation of Devices and Media Runtime Device Software versus the Content Services enabling any Content Service to run on any Device just based on the Device Capabilities, the Media App service mapping, the Device Maker and Content Services restrictions and the User choices only, fully managed in real-time in the cloud. The ecosystem can run on any supported Device running the Media Runtime Device Software, regardless of OS and SoC. Any participant keeps the control it wants, at any time.

[0154] An embodiment is illustrated in Fig. 1 illustrating several elements of the invention. The hatched line indicates what is included in the typically mobile device owned by the user. The Media CDN (content delivery network) is located anywhere in the cloud, i.e., remote from the user device 10 or remote from the apparatus in which the application program is executed, by which the user can control her or his user devices 10. In this embodiment, the mobile device has an interface for interfacing with the Media CDN in the cloud. Fig. 1 shows an example of an ecosystem in particular in the case of providing media content (the same may be obtained, however, for generic services to be provided, and not only media streams to be transmitted). Fig. 1 shows a player device 10 (which may the first processing device 10 in Fig. 2A and 2B or the second processing device in Fig. 3). Each device 10 may control a device hardware 1 (which may be a display device and / or loudspeakers and / or a hardware connection to the loudspeakers. The device 10 may receive a service e.g. media content 51 (which may be divided between 51a and 51b in some examples), which is indicated as 202 in Figs. 2A, 2B, and 3, from a media content distribution network (CDN) 5 (which may be divided between media update CDN 5a, sending 51a, and CDN 5b, sending 51b, in some examples, but this is not strictly necessary; 51a , 51 b, 51 are examples of 202). The assisting system (which may be a cloud backend system) 4 may control the player device 10 (10a, 10b, 10c, 10d), e.g., through content control 38 and / or playback control 34. The player device 10 may include a device SDK / player 1.

[0155] The assisting system 4 may control the playback and the reception of the media streams through control commands 34, 48, received through the first remote connection 401 . The media CDN 5 (second processing system in Figs. 2A and 2B, first processing system in Fig. 3) which provides the content is not the same as the cloud backend system (assisting system) 4. The content 51 (51a, 51b, 202) is transported through the third remote connection 403 and not the first remote connection 401 as the control commands 34 and 38 (it may be that they both share the same physical connection, but anyway the content 51 and the control commands 34 and 38 are not in the same session).

[0156] The device SDK / player 2 may provide the media content (e.g., audio and / or video streams) to a content decryption service 28, which can therefore decrypt the media content 27 (combing from 51 , 51a, 51 b, 202). The decrypted media content 29a may therefore be provided to a hardware media Tenderer unit 1. The hardware media Tenderer unit 1 may include the hardware abstraction layer (HAL) 11. The HAL 11 may be provided by an operating system. The HAL 11 may control the rendering of the audio and / or video. The HAL 11 may be controlled by hardware control and / or hardware events 26, e.g. sent by the device SDK / player 2. The device SDK / player 2 may provide the hardware control and hardware events 26 based, in particular, on the control 34 and 38 accepted by the cloud backend system 4 (assisting system). The player device 10 may be connected to a local network (e.g., LAN) 220. The device SDK / player 2 may include a web browser 21. The device SDK / player 2 may include a media player 22. The device SDK / player may contain a cache 23 (e.g. in the media player 22). The device SDK / player 2 may contain a group playback entity 24. The device SDK / player 2 may contain an updater 25, which may receive updated media content from the update CDN 5a (5). As can be seen from Fig. 1 , the player device 10 (first processing system) may receive content control 38 (e.g. content control commands), which may include, for example, information on queues or more in general on content 37. Fig. 1 shows many queues 37, because it may be possible that information on multiple queues 37 is sent for multiple, different devices 10 (see also below). The content control 38 may provide information on the media content to be rendered (e.g. through the information on the queues or more in general of the content 37), e.g. together with address (e.g., URL address) of where the media content 51 (51a, 51b, 202) is to be retrieved (e.g. address of locations associated to the CDN 5. The cloud backend (assisting system) 4 may transmit playback control commands 34. The playback control 34 may include, for example, well- known commands, such as “PLAY” “PAUSE”, “STOP”, “FAST FORWARD”, “REWIND”. The cloud backend (assisting system) 4 may therefore include a playback manager 33 and a device manager 35 which send the playback control 34, and the content control commands 38 respectively. It is to be noted that the device manager 35 may substantially have knowledge on the properties of the player device 10. The device manager 35 may have a virtual replica of the player device 10 and may therefore have the substantial control of the operations of the player device 10. Both the playback manager 33 and the device manager 35 may be controlled through control commands 31 and 32 respectively. The playback control 31 may be received from a user application 3, which may be executed in a user equipment (e.g., a smart phone, mobile phone, tablet, assigned computer) of property of the user and / or may be integrated in the player device 10 (despite Fig. 1 suggesting that the user equipment and the player device 10 are not integrated in the same device, but they could be). The control device setup commands 32 may command the device management 35 on the groups to be formed between the device 10 and other devices (e.g., devices 10a, 10b, 10c, 10d) (it will be shown that the groups to be formed can be, in some examples, identified by the queues 37 to be reproduced). Therefore, the particular group may be defined by the user through the UE 3.

[0157] The UE 3 may be provided e.g. by the cloud backend system (assisting system) 4 (or by an app store, such as Google Play Store, e.g. in a certified manner) to the user equipment or other device for controlling the player device 10 (and more in general all the player devices 10a-10d). The application 3 may present a user interface for obtaining settings to render content.

[0158] Even if between the UE 3, the cloud backend system (assisting system) 4, the player device

[0159] 10 (e.g. first processing device 10) and the media CDN 5 (e.g. second processing device) it is shown that the arrows are mostly directed towards one direction only, it is to be understood that these arrows mostly refer to the directions of the most relevant commands. It is notwithstanding possible that feedback information may be transmitted in any different direction. For example, the UE 3 may receive feedback from the cloud backend (assisting system) 4 (e.g. to reproduced in a user interface), which in turn may receive the feedback from the player device 10. Also, the direction of the lines 51a and 51 b could be understood bidirectional, since one device (e.g. the primary device 10a, see below) may be the device which requests the transmission to the media CDN 5 and keep some contacts with the media CDN 5 (e.g. by sending requests relating to which representation of adaptation set of the media content to receive, e.g. in response to the reception of a manifest and based on the monitoring of the reception speed, and so on).

[0160] Subsequently, further embodiments of the ecosystem are described. A “final” or endpoint device is the device HW that typically consists of a replay functionality such as a loudspeaker or a video screen and some interface buttons such as stop buttons, pause buttons, etc. This “device HW’ interfaces with a “device SDK / player”. This functional unit “device SDK / player” interfaces, on the one hand, with a media provider or, generally, with a distributed content delivery network (media App CDN). The “device SDK / player” interfaces with the cloud backend 4. There exists an application on the user’s mobile (or stationary) device 3 such as the user’s mobile phone, tablet or even a user’s stationary computer. This app on the mobile device 3 mainly interfaces with the cloud backend system 4 and interfaces, via the onboarding functionality, with the device SDK / player unit. However, the app to be stored, for example, on the user’s mobile device 3 or on the user’s stationary computer 3 does not interface with the final playback device 10 or “device HW” 1. This app does not interface with the media app content delivery network 5 that comprises, for example, a streaming platform such as Netflix, Spotify, Amazon Prime, etc.

[0161] Seen from the perspective of the cloud backend system 5, the cloud backend system 5 interfaces on the one hand, with the app on the user’s mobile 3 or stationary device 3, with the media app content delivery network, and with the device SDK / player. However, the cloud backend system 3 does not interface directly to the actual replay device or “device HW’ 1. In the cases in which the cloud backend system 4 is distributed in a cloud, this functional unit maintains, among others, several databases that include, among others, “Live Services” functionalities that represents the actual interface between the media app content delivery network (media app CDN) 5 and the cloud backend system 4 (assisting system). The cloud ecosystem allows a distributed Communication between the Content Data and the Control Data. It is preferred that the Cinemo cloud backend only communicates with the media app content distribution network with respect to control data that only require a low volume of data transmission. On the other hand, the actual content such as video or audio content is directly transmitted from the distributed content delivery network (CDN) to the actual hardware device via the device SDK / player functionality which is a software program stored on a storage of the actual player device. This distributed transmission of content data on the one hand and control data on the other hand has the advantage of only having low volume traffic in the Cinemo cloud backend. The high volume traffic only occurs between the content delivery network such as the streaming provider and the device SDK / player unit stored in a storage of the end point device including the “device HW” instance.

[0162] The controlled situation in that the user’s access to the content delivery network or streaming provider is only done via the Cinemo cloud backend is advantageous for the content provider, since the content provider can trust the Cinemo cloud backend and can also trust the hardware end point device, since all the replay functionalities are controlled by the device SDK / player instance that tightly cooperates with the Cinemo cloud backend. In case of a security breach the Cinemo cloud backend can easily bar the compromised device or a class or group of compromised devices (e.g., from the same producer or distributor or ...) by building and maintaining a corresponding “no go” database.

[0163] A specific functionality provided to the user of the hardware player is to also press buttons on the loudspeaker or the video screen device giving the user the impression that the user has the full control over the replay functionalities, although these Hardware Abstraction Layer (HAL) events are not forwarded directly to the streaming website, but only via the Cinemo cloud backend. Embodiments of the Cinemo cloud backend make sure that correct reports back to the artist with respect to Copyrights, etc. are assured by the Cinemo cloud backend. Further functionalities regarding the full control for the content provider such as databases stored with user data etc. are used.

[0164] Further embodiments relate to an apparatus, method or computer program for operating any element of the ecosystem as described above, or to an apparatus, method or computer program for communicating between any two elements of the ecosystem as described above, or to an apparatus, method or computer program for operating the ecosystem comprising at least two elements of the ecosystem as described above, or to an apparatus, method or computer program of one of the preceding embodiments, wherein an element of the ecosystem is one element of a group of elements comprising: User Devices, Content Services, Apps, Management Services, Content Decryption Service, Device Hardware, Device SDK Player, Media CDN, Mobile / Desktop Applications, Device Management, Playback Management and Device Queues as described.

[0165] Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Analogously, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus.

[0166] Depending on certain implementation requirements, embodiments of the invention can be implemented in hardware or in software. The implementation can be performed using a digital storage medium, for example a floppy disk, a DVD, a CD, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed. Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0167] Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may for example be stored on a machine readable carrier. Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier or a non-transitory storage medium. In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer. A further embodiment of the inventive methods is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein. A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may for example be configured to be transferred via a data communication connection, for example via the Internet. A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein. A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein. In some embodiments, a programmable logic device (for example a field programmable gate array) may be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods are exemplarily performed by any hardware apparatus.

[0168] The above described embodiments are merely illustrative for the principles of the present invention. It is understood that modifications and variations of the arrangements and the details described herein will be apparent to others skilled in the art. It is the intent, therefore, to be limited only by the scope of the impending patent claims and not by the specific details presented by way of description and explanation of the embodiments herein.

Claims

Claims1. User device (100) comprising: a device hardware (102) comprising: a storage (102c) for detection information, wherein the storage (102c) comprises a plurality of storage entries (103a, 103b, 103c), each storage entry (103a, 103b, 103c) having a specific detection information item associated with a data instance identification of a specific data instance (401 ,402, 403, 404, 405) of a plurality of different data instances (401 , 402, 403, 404, 405), and a monitoring device (102a) for monitoring an environment of the user device (100); and a media runtime device software (110) comprising a media interface (114) to one or more data instances (401 , 402,403, 404, 405), and being configured for processing a monitoring result of the monitoring device (102a) to obtain a detected detection information item of a plurality of different detection information items, for accessing the storage (102c) to obtain the data instance identification of the specific data instance (401 , 402,403, 404, 405) associated with the detected detection information item, and for initiating (808) a connection to the specific data instance (401 , 402, 403,404, 405) associated with the detected detection information item using the data instance identification of the specific data instance (401 , 402, 403, 404, 405).

2. User device (100) of claim 1 , wherein the media runtime device software (102) comprises a control interface (112) to a cloud backend (200), and wherein the media runtime device software (110) is configured to receive commands for controlling the device hardware (102) via the control interface (112), and to receive or transmit content via the media interface (114) in response to a corresponding command received via the control interface (112).

2. User device (100) of claim 1 , wherein the media runtime device software (110) is configured to receive, via the control interface (112), content metadata (750) comprising information identifying the data instance and an access information, and wherein the media runtime device software (110) is configured to access (752) the data instance (400) using the received access information via the media interface (114).

3. User device (100) of claim 1 or 2, wherein the data instance (400) is a content provider, wherein the media runtime device software (110) is configured, in the initiating a connection to the specific data instance (401 , 402, 403, 404, 405) associated with the detected detection information item, to receive playback metadata identifying intended content data for playback via the control interface (112), to send (750) the playback metadata to the content provider via the media interface (114), and to receive (756) the content for playback from the content provider via the media interface (114).

4. User device (100) of one of the preceding claims, wherein each one of the plurality of data instances (401 , 402, 403, 404, 405) comprises a user-specific communication interface for communicating with the user device, wherein the media runtime device software (110) is configured, in the initiating a connection to the specific data instance (401 , 402, 403, 404, 405) associated with the detected detection information item, to send (752) a request for a user-specific content to the specific data instance, to play (754) the user-specific content; to receive (768, 766, 776) a user reaction to the user-specific content; andto communicate (772, 774) with the user-specific communication interface in response to the user reaction.

5. User device (100) of claim 4, wherein the media runtime device software (110) is configured, in receiving (768, 766, 776) a user reaction to the user-specific content, to receive (768) a user input via a user interface (102d) included in the device hardware (102), or to send (758) information on the user-specific content to the cloud backend (200), and to receive (766) a user reaction to the user-specific content, or to directly communicate (776) with an apparatus (300) for performing a user application to obtain a user reaction to the user-specific content.

6. User device (100) of one of the preceding claims, wherein the media runtime device software (110) is configured to log into a cloud backend (200); to receive (628, 834) an updated detection information in association with a specific data instance (401 , 402, 403, 404, 405) and an identification of the specific data instance; and to overwrite (836), in the storage entry (103a, 103b, 103c) for the specific data instance of the plurality of different data instances (401 , 402, 403, 404, 405) a stored detection information with the updated detection information.

7. User device (100) of claim 6, wherein the media runtime device software (110) is configured to receive, from the cloud backend (200) a request for a list of data instances stored in the plurality of storage entries (103a, 103b, 103c); to access the storage (102c) to generate the list and to send (834) the list; and to receive a request to update one or more storage entries for one or more data instances included in the list.

8. User device (100) of one of the preceding claims, wherein the monitoring device (102a) comprises a microphone and the detection information is a key word or a wake word, or wherein the monitoring device (102a) comprises a camera and the detection information is an image or video of a key sign or key gesture.

9. User device (100) of one of the preceding claims, wherein the device hardware (102) is configured to transition into a sleep mode, in which the monitoring device (102) is active, and in which the media runtime device software (110) is configured for processing the monitoring result of the monitoring device (102a), and wherein the media runtime device software (110) is configured to transition the device hardware (102) into an active mode, when one of the plurality of different detection information items stored in the storage (102c) is detected in the monitoring result.

10. User device (100) of one of the preceding claims, wherein the monitoring device (102a) comprises a microphone and the detection information is a key word, and wherein the media runtime device software (110) is configured for processing the monitoring result of the monitoring device (102a) using one or more trained neural networks.

11. User device of one of the preceding claims, wherein the device hardware (102) is configured to only replay content identified via a command received via the control interface (112) and received via the media interface (114).

12. User device of one of the preceding claims, wherein the media runtime device software (110) is configured to have a web browser functionality, a media player functionality, or a content decryption functionality, wherein the functionalities are fully controlled by control information received from the cloud backend via the control interface (112).

13. User device of one of the preceding claims, wherein the media runtime device software (110) is configured to retrieve (750) a unique resource locator (URL) for a specific remote data instance (401 to 405) via the control interface (112), to interface (752, 756) with the specific remote data instance using the playable URL via the media interface (114), and to retrieve user-specific content data defined by the unique resource locator via the media interface (114).

14. Cloud backend (200) comprising: a detection information library (208a) for storing, for each one of a plurality of data instances (401 , 402, 403, 404, 405), a detection information item associated with an identification of the corresponding data instance; a user device interface (202) for interfacing with a user device (100); and a communication interface (204) for interfacing with the plurality of data instances (401 , 402, 403, 404, 405), wherein the cloud backend (200) is configured to receive a message (715) from the user device via the user device interface (202) indicating that the user device has detected a call for a specific data instance of the plurality of data instances (401 , 402, 403, 404, 405) stored in the detection information library (208a), and to provide a message (750) to the user device interface (202) for enabling the user device to access user-specific media data at the specific data instance of the plurality of data instances (401 , 402, 403, 404, 405) stored in the detection information library (208a).

15. Cloud backend (200) of claim 14, further comprising a communication interface (204) for interfacing with the plurality of data instances (401 , 402, 403, 404, 405), wherein the cloud backend is configured to communicate to the plurality of data instance (400) via the communication interface (204) to manage an authorization of a user of the user device, to receive data from the data instance, or to send data to the data instance.

16. Cloud backend (200) of claim 15, configured to receive (822) a message from a specific data instance of the plurality of data instances (401 , 402, 403, 404, 405) regarding an updated detection information, to access (826) the detection information library (208a) and to store the updated detection information in association with the specific data instance, and to activate (828) an update service for a user device logging into the cloud backend (200).

17. Cloud backend of claim 16, wherein the cloud backend (200) is configured to perform (824) an authorization of the message from a specific data instance of the plurality of data instances (401 , 402, 403, 404, 405) regarding an updated detection information and to only activate (828) the update service, when the message is successfully authorized in the performing (824) the authorization.

18. Cloud backend of claim 16 or 17, wherein the cloud backend (200) is configured to perform (830) a login detection of a user device, and to send (834) a message comprising the updated detection information to the user device.

19. Cloud backend of claim 18, wherein the cloud backend (200) is configured, subsequent to the performing (830) the login detection of the user device, to ask (832) the user device for a list of data instances having detection information stored in the user device; to check (832) the list of data instances after reception from the user device; and wherein the sending (834) the message comprising the updated detection information is only performed, when the specific data instance is on the list.

20. Cloud backend of one of the claims 14 to 19, further comprising an application interface (206) for interfacing with a user application transmitting information on a user input.

21. Cloud backend of claim 20, wherein the cloud backend is configured to receive, via the application interface (206), a user authentication request (700), to check whether the user is authorized using an authorization data base, and to send (705) a user token via the application interface (206) in case the user is authorized.

22. Cloud backend of claim 20 or 21 , wherein the cloud backend is configured for receiving a request (702) for one or more user-authenticated data instances via the application interface (206), to check (704, 706, 708) if the user is authorized for theplurality of data instances using a data instance database and to send (710), via the application interface (206) an information about the one or more user authenticated data instances.

23. Cloud backend of one of claims 14 to 22, wherein the cloud backend is configured to receive, via the user device interface (202), a request (715) for a specific data instance, to send, via the communication interface (204), user-specific credentials (718) for the selected data instance which are retrieved (716) from a credential database in the cloud backend for a user authentication at the selected data instance; and to receive (720), via the communication interface (204), an authentication result.

24. Cloud backend of one of the preceding claims, wherein the cloud backend is configured to send, in case of a positive authentication result, a content metadata request (722) to the specific data instance via the communication interface (204), and to receive (724), via the communication interface (204), the content metadata on the user-specific content at the specific data instance.

25. Cloud backend of one of the claims 14 to 24, wherein the cloud backend is configured to send, via the application interface (206) or via the user device interface (202), a selected data instance information (725) and to receive, via the application interface (206) or via the user device interface (202), a connection activation confirmation (748).

26. Cloud backend of one of the preceding claims, wherein the cloud backend is configured to send (750), via the user device interface (202), metadata for the selected content to the selected device.

27. Cloud backend of one of the claims 10 to 20, wherein the cloud backend is configured to receive a request (620) for a data instance authorization from the user via the application interface (206) or the user device interface (202); look-up an authorization method required by the data instance in a data instance database in the cloud backend;send an authorization request (622) using the authorization method via the communication interface (204); receiving an invitation (623) from the data instance via the communication interface (204) asking for credentials and / or consent and forwarding the information on the credentials and / or consent via the application interface (206); receiving the credentials (604) via the application interface (206) and storing the credentials in a credential database in the cloud backend; receiving (625) an access token for the user via the communication interface (204) and storing (626) the access token in the cloud backend; and sending (628) a detection information for the data instance to the user device (100) via the user device interface (202).

28. Apparatus (300) for performing a user application, comprising: a user interface (302) for receiving an input from a user (99); a cloud backend interface (304) for communicating with a cloud backend (200), wherein the apparatus is configured to communicate, via the user interface (300), to receive user commands regarding a selection of a plurality of data instances for a communication, to receive, via the cloud backend interface (304), a detected data instance message (725) indicating a specific data instance of a plurality of data instances (401 , 402, 403, 404, 405), and to send, via the cloud backend interface (304), a data instance confirmation message (748) regarding the specific data instance of a plurality of data instances (401 , 402, 403, 404, 405).

29. Apparatus of claim 28, wherein the apparatus is configured to receive a data instance authorization request (620) via the user interface (302),to send a data instance authorization message (621) via the cloud backend interface (304), to receive a credentials input request (623) for the data instance from the cloud backend interface (304), to receive the credentials via the user interface (302) and to send the credentials via the cloud backend interface (304); and to receive (624) a confirmation via the cloud backend interface (304) and to output the confirmation (627) via the user interface (302).

30. Apparatus of claim 28 or 29, wherein the apparatus is configured to receive a login request (603) from the user interface (302), to send an authorization request (604) via the cloud backend interface (304); and to receive a user token (605) including a user ID or device ID via the cloud backend interface (304).

31. Apparatus of one of claims 28 to 30, wherein the apparatus is configured to request (702) information on one or more data instances for a user identification of the user via the cloud backend interface (304); and to receive (710) a list of one or more allowed data instances for the user via the cloud backend interface (304).

32. Apparatus of one of claims 28 to 31 , wherein the apparatus is configured to receive a data instance confirmation (746) via the user interface (302) subsequent to the receiving, via the cloud backend interface (304), a detected data instance message (725) indicating a specific data instance of a plurality of data instances (401 , 402, 403, 404, 405), and wherein the sending, via the cloud backend interface (304), the data instance confirmation message (748) regarding the specific data instance of a plurality of datainstances (401 , 402, 403, 404, 405) is only performed when the data instance confirmation (746) is received via the user interface (302).

33. Apparatus of one of claims 22 to 28, wherein the apparatus is configured, during a session at a specific data instance, to receive (760, 766) a content provided by the specific data instance from the cloud backend interface (304) or via a user device interface (303), to render the content via the user interface (302); to receive (762) a user reaction to the content via the user interface (302); and to communicate (764) the user reaction via the cloud backend interface (304) or the user device interface (303).

34. Method of operating a user device comprising: a device hardware (102), and a media runtime device software (110) configured for controlling the device hardware (102), comprising: storing, in a storage (102c) comprising a plurality of storage entries (103a, 103b, 103c), specific detection information items associated with data instance identifications of a plurality of different data instances (401 , 402, 403, 404, 405); processing a monitoring result to obtain a detected detection information item of a plurality of different detection information items, accessing the storage (102c) to obtain the data instance identification of the specific data instance (401 , 402, 403, 404, 405) associated with the detected detection information item, and initiating a connection to the specific data instance (401 , 402, 403, 404, 405) associated with the detected detection information item using the data instance identification of the specific data instance (401 , 402, 403, 404, 405).

35. Method of operating a cloud backend (200) Cloud backend (200) comprising:storing, for each one of a plurality of data instances (401, 402, 403, 404, 405), a detection information item in associating with an identification of the corresponding data instance; interfacing with a user device (100); interfacing with the plurality of data instances (401, 402, 403, 404, 405), receiving a message (715) from the user device indicating that the user device has detected a call for a specific data instance of the plurality of data instances (401 ,402, 403, 404, 405); and providing a message (750) to the user device interface (202) for enabling the user device to access user-specific media data at the specific data instance of the plurality of data instances (401 , 402, 403, 404, 405).

36. Method of operating a user application (300), comprising: receiving an input from a user (99) via a user interface (302); communicating with a cloud backend (200) via a cloud backend interface (304); communicating, via the user interface (300) to receive user commands regarding a selection of a plurality of data instances for a communication, receiving, via the cloud backend interface (304), a detected data instance message (725) indicating a specific data instance of a plurality of data instances (401, 402,403, 404, 405), and send, via the cloud backend interface (304), a data instance confirmation message (748) regarding the specific data instance of a plurality of data instances (401 , 402, 403, 404, 405).

37. Computer program for performing, when running on a computer or a processor, the method of one of claims 30 to 32.

38. Cloud system comprising:one or more user devices as defined in one of claims 1 to 13; a cloud backend as defined in one of the claims 14 to 27; and one or more apparatuses for performing a user application of one of claims 28 to 33; wherein the cloud backend interface (304) of the one or more user applications (300) is connected to the application interface (206) of the cloud backend, wherein the control interface (112) of the one or more user devices (100) is connected to the user device interface (202) of the cloud backend, wherein the communication interface (204) of the cloud backend is connected to a remote data instance for control purposes only; and wherein the media interface (114) of the one or more user devices (100) is connected to the remote data instance (400) for the purpose of downloading or uploading data.

Images