Keyword recognition apparatus, keyword variant generation apparatus, media processor and related methods

Abstract

A keyword recognition apparatus (2000), comprising: an input interface (102d) for receiving an input audio data (2002); a trained acoustic model (2008) configured for processing the input audio data (2002) and for outputting a predicted sequence of phonemes (2014); a memory (102c) for storing a plurality of variant keyword sequences of phonemes (2020) for a specified keyword; a keyword matcher (2016) for detecting a match between the predicted sequence of phonemes (2014) and a variant keyword sequence of phonemes of a plurality of variant keyword sequences of phonemes (2020); and an output interface (2024) for performing an action (2026) in response to a detection of the match (2022).

Description

[0001] Keyword Recognition Apparatus, Keyword Variant Generation Apparatus, Media Processor and Related Methods

[0002] Specification

[0003] The present invention relates to audio data processing, and particularly to speech processing including keyword recognition comprising the generation of keyword variants and the recognition of a keyword using different keyword variants.

[0004] There are many electrical devices that allow a user to interact with the device via speech. Such devices cover a wide range of technologies, such as smart phones, watches, navigation systems, home entertainment systems - in other words, any device that contains a speaker and a processing unit has the potential to have a voice-activated system integrated into it. Devices with voice-activated systems typically have multiple functionalities that can be controlled by a user’s voice, for example, in a non-exhaustive list: waking a device from a sleep mode, accessing and controlling applications on the device, such as opening a music application and playing a song, or interacting with an A. I. companion, such as Siri or Alexa.

[0005] The use of a specific keyword, often referred to as a wake word, allows a user to activate a function on a device that is associated with the given keyword. For this, a device will typically continuously record the audio of its surrounding on a microphone and process the data on a processor. Upon “hearing” a keyword, meaning upon processing the audio data and recognising a sequence that the device associates with a keyword, the device will take an action.

[0006] One of the largest difficulties for the implementation of keyword activation is dealing with the many accents that can be found around the world. Keyword activation is based on detecting the sounds that are associated with a keyword, therefore, if, for example, two different people try to voice-activate the same device, but they pronounce the same keyword differently from each other, then these different pronunciations still need to be recognised as the same keyword by the device. But not only accents have to be accounted for, also the different speeds at which different people speak

[0007] It is an objective of the present disclosure to provide an improved concept for keyword handling.

[0008] The objective is achieved by a keyword recognition apparatus in accordance with claim 1 , a keyword variant generation apparatus in accordance with claim 18, a media processor in accordance with claim 22, a method of operating a keyword recognition apparatus in accordance with claim 26, a method of operating a keyword variant generation apparatus in accordance with claim 27, a method of operating a media processor in accordance with clam 28, or a computer program of claim 29.

[0009] A keyword recognition apparatus implementing the present invention comprises an input interface for receiving an input audio data, a trained acoustic model configured for processing the input audio data and for outputting a predicted sequence of phonemes, a memory for storing a plurality of variant keyword sequences of phonemes for a specified keyword, a keyword matcher for detecting a match between the predicted sequence of phonemes and a variant keyword sequence of phonemes of a plurality of variant keyword sequences of phonemes, and an output interface for performing an action in response to a detection of the match.

[0010] According to an example embodiment of a keyword recognition apparatus, the memory is configured to store a second plurality of variant keyword sequences of phonemes for a second specified keyword, different from the specified keyword, wherein the keyword matcher is configured to compare the predicted sequence of phonemes to the plurality of variant keyword sequences of phonemes and the second plurality of variant keyword sequences of phonemes, and to detect the match for a matching specified keyword, the matching specified keyword having a variant keyword sequence of phonemes of its associated first or second plurality of variant keyword sequence of phonemes being in a predefined relation to the predicted sequence of phonemes.

[0011] According to an example embodiment of a keyword recognition apparatus, the output interface is configured to perform a second action for the second specified keyword, the second action being different from the first action for the specified keyword.

[0012] According to an example embodiment of a keyword recognition apparatus, the output interface is configured to perform either the first or the second action depending on whether the matching specified keyword is the specified keyword or the second specified keyword.

[0013] According to an example embodiment of a keyword recognition apparatus, the output interface is configured to determine the action so that the specified keyword is identified, or to determine the second action so that the second keyword is identified. According to an example embodiment of a keyword recognition apparatus, the keyword matcher is configured to compare the predicted sequence of phonemes to the plurality of variant keyword sequences of phonemes in parallel to comparing the predicted sequence of phonemes to the second plurality of variant keyword sequences of phonemes.

[0014] According to an example embodiment of a keyword recognition apparatus, the keyword matcher is configured to perform the comparison using a separate state machine for each variant keyword sequence of phonemes for the comparison of the predicted sequence of phonemes to the plurality of variant keyword sequences of phonemes and to the second plurality of variant keyword sequences of phonemes.

[0015] According to an example embodiment of a keyword recognition apparatus, an output of a state machine comprises the variant keyword sequence of phonemes with which the state machine is associated.

[0016] According to an example embodiment of a keyword recognition apparatus, the trained acoustic model comprises a trained neural network block for receiving the input audio data and for outputting a probability of predictions, and a prediction decoder for decoding the plurality of predictions to obtain the prediction sequence of phonemes.

[0017] According to an example embodiment of a keyword recognition apparatus, the input interface is configured to receive, as the input audio data, an audio file, or wherein the input interface is configured to receive a stream of audio data sequences, and wherein the input interface is configured to restructure an input audio data file or a stream of audio data sequences into a sequence of blocks of a certain length.

[0018] According to an example embodiment of a keyword recognition apparatus, the input audio data is structured to comprise of a plurality of audio data blocks, each with a length of 10 ms, wherein the trained neural network block is configured to make a plurality of predictions on every second audio data block, wherein each prediction of the plurality of predictions is associated with a different phoneme, wherein a prediction of the plurality of predictions comprises a likelihood, and the likelihoods of the plurality of predictions sum to 1.

[0019] According to an example embodiment of a keyword recognition apparatus, the prediction decoder is configured to perform an argmax decoding among the plurality of predictions to obtain the predicted phoneme. According to an example embodiment of a keyword recognition apparatus, the prediction decoder is configured to filter out the predicted phoneme, if the predicted phoneme comprises of the same phoneme as the predicted phoneme of the previous audio data block for which a plurality of predictions was made.

[0020] According to an example embodiment of a keyword recognition apparatus, the memory is configured to store a plurality of three or more keyword sequences of phoneme variants for each keyword.

[0021] According to an example embodiment of a keyword recognition apparatus, the keyword recognition apparatus further comprises a voice activity decoder, wherein the trained acoustic model is configured to receive a signal from the voice activity decoder, wherein the trained acoustic model is configured to change its behaviour in accordance to the received signal.

[0022] A keyword variant generation apparatus implementing the present invention comprises an input interface for receiving a keyword, an analyser for analysing the keyword to obtain an analysed phoneme sequence, a keyword variant producer for producing a plurality of keyword sequences for the analysed phoneme sequence, and an output interface for outputting the plurality of keyword sequences.

[0023] According to an example embodiment of a variant generation apparatus, the analyser is configured to generate the analysed phonemes sequence.

[0024] According to an example embodiment of a variant generation apparatus, the keyword variant generation apparatus is configured to use a phoneme confusion statistic.

[0025] According to an example embodiment of a variant generation apparatus, the keyword variant generation apparatus is configured to generate the confusion statistics, or a confusion matrix according to a latent representation of a training data set comprising text-aligned speech samples comprising audio input and corresponding sequence of truth labels.

[0026] A media processor implementing the present invention comprises a keyword recognition apparatus, a microphone for capturing an audio signal wherein the media processor is configured to derive the input audio sequence received by the input interface from the audio signal, and a processor configured for performing a processing move depending on the action performed by the output interface. According to an example embodiment of a media processor, the processor is configured to have a sleep mode or power mode, and the microphone is active in the sleep mode, and the processor is configured for changing from the sleep mode to the power mode in response to the action performed by the output interface.

[0027] According to an example embodiment of a media processor, the keyword receiving apparatus is configured to store a second plurality of variant keyword sequences of phonemes for a second specified keyword, different from the specified keyword, wherein the keyword matcher is configured to compare the predicted sequence of phonemes to the plurality of variant keyword sequences of phonemes and the second plurality of variant keyword sequences of phonemes, and to detect the match for a matching specified keyword, the matching specified keyword having a variant keyword sequence of phonemes of its associated first or second plurality of variant keyword sequence of phonemes being in a predefined relation to the predicted sequence of phonemes. The output interface is configured to perform a second action for the second specified keyword, the second action being different from the first action for the specified keyword, and the processor is configured to generate a first message to a first external data instance in response to the second message for the second different external data instance in regard to the second action.

[0028] According to an example embodiment of a media processor, the output interface of the keyword variant generation apparatus is connected to the memory of the keyword recognition apparatus.

[0029] A method of operating a keyword recognition apparatus implementing the present invention comprises receiving, to an input interface, an input audio data, configuring a trained acoustic model for processing the input audio data and for outputting a predicted sequence of phonemes, storing, on a memory, a plurality of variant keyword sequences of phonemes for a specified keyword, detecting, via a keyword matcher, a match between the predicted sequence of phonemes and a variant keyword sequence of phonemes of a plurality of variant keyword sequences of phonemes, and performing, via an output interface, an action in response to a detection of the match.

[0030] A method of operating a keyword variant generation apparatus implementing the present invention comprises receiving, via an input interface, a keyword according to claim, analysing, via an analyser, the keyword to obtain an analysed phoneme sequence, producing, via a keyword variant producer, a plurality of keyword sequences for the analysed phoneme sequence, and outputting, via an output interface, the plurality of keyword sequences. The present invention is advantageous in that the separation of the task of keyword recognition into a moderately complex neural network task for generating the predicted sequence and a quite efficient deterministic task for matching the predicting sequence over explicit keyword variant sequences has shown to be an optimum compromise for an overall efficient implementation with an excellent recognition capability. The neural network can be made with a limited complexity since the neural network only has to provide a predicted phoneme sequence without any knowledge on the currently searched keyword. This saves a lot of complexity compared to the case where a fully trained neural network is used that is explicitly trained for a specific keyword.

[0031] The separation of the actual keyword detection from the neural network also makes it possible to use the same fully trained neural network for different keywords. This feature is not only useful when a plurality of keywords has to be detected with one and the same neural network, but also allows that the user selects their own keyword and nevertheless does not have to train a neural network with the new keyword. The user simply selects their own personal keyword and uses exactly the same trained neural network for the new private keyword.

[0032] The “keyword reduction” is done in the deterministic matching step that is typically highly efficient and can be easily implemented, since the “interface” between the neural network and the matching procedure are speech phonemes. The output of the neural network block is the predicted sequence of phonemes, and the “treasury” used by the matching step is the multitude of variant keyword sequence of phonemes that form the data basis against which the predicted sequence of phonemes is matched.

[0033] In a preferred embodiment, the neural network comprises a trained neural network block and a subsequently connected prediction decoder. The trained neural network block can be a straightforward, off the shelve and readily available module that generates, for each chunk of audio data, a list of different phonemes with different probabilities. For a plurality of sequential chunks, a sequence of such lists is obtained, that can be further processed by the prediction decoder to bring this sequence of such lists into a predicted phone sequence that can then be used by the matching instance. Thus, the prediction decoder performs a conditioning of the output of universal neural network blocks into phoneme data that can then be used by the keyword matcher, which is the only instance that has knowledge on the actually used one or more keywords.

[0034] Preferably, a device is able to recognise multiple keywords, each leading to a different action upon being recognised by the device. This becomes very useful one a device that has access to multiple functionalities, as different keywords can be associated with different actions, for example, on a smart phone, the keyword “YouTube” can be used to open the YouTube application and the keyword “Spotify” can be used to open the Spotify application.

[0035] Advantageously, the device is able to recognise many different variants of the same keyword, which is especially useful for implementations that allow users to generate their own keywords for a device to recognise, as the variants also are taken into account.

[0036] Preferably, the inventive keyword recognition apparatus is configured to recognise a plurality of variants of not only one, but more than one different keyword. The inventive keyword generation apparatus is configured for generating a plurality of keyword variants for a user specified keyword or more than one keywords that can then be used by the keyword generation apparatus.

[0037] A method of operating a media processor implementing the present invention comprises operating a keyword recognition apparatus, capturing, via a microphone, an audio signal herein the media processor is configured to derive the input audio sequence received by the input interface from the audio signal, and a processor configured for performing a processing move depending on the action performed by the output interface.

[0038] A computer program implementing the present invention for carrying out any one of the methods of operating a keyword recognition apparatus, of operating a keyword variant generation apparatus, or operating a media processor.

[0039] Preferred embodiments of the present invention are subsequently disclosed with respect to the accompanying figures, in which:

[0040] Fig. 1 illustrates a sequence of steps performed by the keyword variant generator in accordance with an embodiment;

[0041] Fig. 2a illustrates a sequence of steps performed by the keyword recognition apparatus in accordance with an embodiment;

[0042] Fig. 2b illustrates a sequence of steps performed by the keyword recognition apparatus in accordance with an embodiment; Fig. 3a illustrates a sequence of steps performed by keyword matcher in accordance with an embodiment;

[0043] Fig. 3b illustrates a sequence of steps performed by the trained acoustic model in accordance with an embodiment;

[0044] Fig. 3c illustrates an example output of the keyword recognition apparatus in accordance with an embodiment;

[0045] Fig. 3d illustrates an example for a table depicting a label for each phoneme;

[0046] Fig. 4a illustrates a media processor in accordance with an embodiment;

[0047] Fig. 4b illustrates a sequence of steps performed by the keyword recognition in accordance with an embodiment;

[0048] Fig. 5 illustrates an implementation of a cloud ecosystem in accordance with an embodiment;

[0049] Fig. 6a illustrates a first portion of a sequence of messages between different entities for the purpose of playing content;

[0050] Fig. 6b illustrates a second portion of a sequence of messages for the purpose of playing a content and for the purpose of performing control playback during playing.

[0051] Fig. 7a illustrates an embodiment of the user device with a detection item storage;

[0052] Fig. 7b illustrates a specific embodiment of the storage of Fig. 7a;

[0053] Fig. 7c illustrates a sequence of procedures performed at least in part by the user device;

[0054] Fig. 8 illustrates a sequence of steps to be performed for updating a certain detection information in an embodiment; and

[0055] Fig. 9 illustrates an implementation of the cloud backend with a detection item storage. Various aspects are now described with reference to the drawings. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more aspects. It may be evident to a skilled person in the art that the various aspects may be practised without these specific details. Aspects of the described examples can be combined with one another, unless aspects are incompatible with one another and such combinations are excluded.

[0056] The features, nature, and advantages of the present invention will become more apparent from the detailed description set forth below when taken in conjunction with the drawings in which like reference characters identify like elements correspondingly throughout the specification and drawings. Elements that are the same, elements that are similar, or elements that have the same functionality may be given the same or similar reference characters; however, a repeated description of elements with the same or similar reference characters, or with the same name, is typically omitted. Descriptions of elements, that have the same or similar reference characters, or have the same name, can be interchanged with one another.

[0057] Fig. 1 shows a block diagram for a keyword variant generator 1000, the keyword variant generator 1000 comprises an input interface 1004, an analyser 1008, a keyword variant producer 1012 and an output interface 1016. The input interface 1004 is connected to the analyser 1008, the analyser 1008 is connected to the keyword variant producer 1012 and the keyword variant producer 112 is connected to the output interface 1016. Subsequently, the functionality is explained. The general idea for a keyword variant generator 1000 is for a user to be able to input a keyword and for the keyword variant generator 1000 to then create a plurality of variance phenome sequences 1018 that are all very similar to the phoneme sequence that describes the keyword 1002 that was input by the user. The keyword 1002 that is input by a user is received by an input interface 1004. The input interface 1004 further passes this keyword 1006 onto the analyser 1008. The analyser 1008 translates the keyword 1006 into a phenome sequence, here referred to as an analysed phenome sequence 1010. The keyword variant producer 1012 takes the analysed phoneme sequence 1010 and uses it to generate a plurality of variance phoneme sequences 1014. The plurality of variance phenome sequences 1014 comprises of any number of phoneme sequences that are all very similar to the analysed phoneme sequence 1010. The plurality of variant phoneme sequences 1014 is generated using confusion statistics. The resulting plurality of variant phenome sequences 1014 are given to the output interface 1016 which outputs a plurality of phenome sequences 1018. It should be noted that a keyword 1002 can be represented as a few or several hundred, up to thousands of variants, depending on the keyword, meaning depending on the length of the keyword and the involved phonemes. The number of variance phoneme sequences in the plurality of variance phoneme sequences 1014 can be reduced, for example by tuning which keyword variants are selected based on a test set. The underlying phoneme translation specifications can be tuned manually based on testing, or in another embodiment even fully specified based on testing. It is also possible, for example, to use confusion statistics on the output layer, evaluated on a generic language corpus.

[0058] The confusion statistics comprise a confusion matrix according to a latent representation of a training data set comprising text aligned speech samples comprising audio input and corresponding sequence of truth labels. In other words, using a large corpus of unseen textline speech samples comprised of audio input in corresponding sequence of ground truth labels, a dxT latent representation is collected for all labels across all time stamps across all samples. Here, d represents the number of canonical phonemes and t represents a time block. These latent representations are filtered by imposing a strict filter based on model likelihood estimates. A distance matric between populations of representations in the latent representations is defined, for example, using the cosine distance between the centroids of different populations. A confusion matrix is defined that represents a likelihood of confusion between populations of ground truth labels. This likelihood is derived based on the distribution of distances between populations of latent representations. For a given input sequence of target labels, acceptable variant sequences are generated by probabilistically substituting characters according to the statistics, for example using a confusion matrix based on latent space statistics, as explained above.

[0059] Fig. 1 depicts a process for a single keyword, but it can be repeated with new keywords, for example a second keyword, which will result in the generation of a second plurality of variant phonemes sequences.

[0060] Fig. 2a shows a block diagram of a keyword recognition apparatus 2000. The keyword recognition apparatus 2000 comprises an input interface 102d, a train acoustic model 2008 comprising a trained neural network block 2010 and a prediction decoder 2012, a keyword matcher 2016, a memory 102c and an output interface 2024. The input interface 102d is connected to the trained acoustic model 2008, a trained acoustic model 2008 is connected to the keyword matcher 2016, the memory 102c is connected to the keyword matcher 2016, and the keyword matcher 2016 is connected to the output interface 2024. Subsequently, the functionality of the keyword recognition apparatus 2000 is explained. An input audio data 2002 is fed to the input interface 102d. The input interface 102d passes on the input audio data 2002 to the trained acoustic model 2008. The trained acoustic model 2008 outputs a predicted sequence of phonemes 2014 and passes them to the keyword matcher 2016. The plurality of variance phoneme sequences 1018, as output by the output interface 1016 of the keyword variant generator 1000 as described in Fig. 1 , are stored on the memory 102c of the keyword recognition apparatus 2000. This means that the memory 102c stores a plurality of variant phoneme sequences from one or more keywords, for example a plurality 2020 and a second plurality 2021 , which are then passed to the keyword matcher 2016. The keyword matcher 2016 compares the predicted sequence of phonemes 2014 with a plurality of variant sequences oh phonemes 2020 for one or more keywords 2020 and passes information about a possible match 2022 to the output interface 2024. Based on the information of a possible match 2022, the input interface 2024 will take an action 2026.

[0061] Fig. 2b shows a block diagram of a keyword recognition apparatus 2000 that is to be understood in the same context as Fig. 2a, further comprising a voice activity detector 2004. The voice activity detector determines when speech is started, and sends a signal 2003 to the trained acoustic model 2008 to activate it when speech is detected. This reduces the computational load, as a voice activity detector 2004 is more lightweight than an acoustic model 2008, but it also provides a defined state for the acoustic model 2008, which is highly relevant when, for example, the trained acoustic model 2008 is an RNN model.

[0062] Fig. 3a shows a block diagram for an example embodiment 4000, where the example embodiment 4000 comprises a trained acoustic model 2008, a keyword matcher 2016 comprising a state machine 1 4002, a state machine 2 4004, state machine 3 4006, state machine 4 4008, a memory 102c, and an output interface 2024. The trained acoustic model 2008 is connected to the state machine 1 4002, the state machine 2 4004, the state machine 3 4006, the state machine 4 4008, the four state machines 4002, 4004, 4006, 4008 are connected to the output interface 2024 and the memory 102c is connected to the keyword matcher 2016. Subsequently, the functionality of the example 4000 is explained. An audio data 2002 is passed to a training acoustic model 2008, which predicts phonemes. Once a certain number of phonemes has been predicted, a predicted sequence of phonemes 2014 is passed to the state machines 4002, 4004, 4006, 4008. In the example 4000, the memory of the keyword matcher 2016 comprises two keywords, a keyword 1 and keyword 2, which each have two variants, resulting in a total of four keyword variants. Each state machine 4002, 4004, 4006, 4008 is assigned a keyword variant and compares that keyword variant to the predicted sequence of phonemes 2014. Each state machine 4002, 4004, 4006, 4008 checks for a possible match 2022, and feeds information about a match to the output interface 2024. According to the information about a match 2022, the output interface 2024 will send out a message for an action 2026. This example shows how the keyword matcher 2016 is able to compare more than one keyword variant to the predicted sequence of phonemes 2014 at a single time. In other examples, in which the memory 102c for the keyword matcher has stored more than two variants for a single keyword and / or more than two keywords, the keyword matcher 2016 requires a higher number of state machines.

[0063] Fig. 3b shows a detailed block diagram for the trained acoustic model 2008, where the trained acoustic model 2008 comprises, a functionality for splitting 4015 an input audio set 2002 into a plurality of audio data blocks, a trained neural network block 2010, which contains forty phoneme blocks, whereby only phoneme block 1 4020, phoneme block 204058 and phoneme block 40 4094 are depicted, a prediction decoder 2012, which comprises of an argmax functionality 4100 and a functionality for checking for a phoneme change 4014. The functionality for into a plurality of audio data blocks 4015 is connected to phoneme block 1 4020, phoneme block 204058 and phoneme block 404098. Phoneme block 1 4020, phoneme block 20 4058 and phoneme block 40 4098 are connected to the argmax functionality 4100. The argmax functionality 4100 is connected to the functionality for checking for a phoneme change 4014. Subsequently, the functionality of a trained acoustic model 2008 is explained. An input audio set 2002 is split into a plurality of audio data blocks 4015, which each have a set length. A single audio data block is passed 4016 to a trained neural network block 2010 of the trained acoustic model 2008, wherein the single audio data block is passed 4016 to each of the forty phoneme blocks, among which block 1 4020, block 20 4058 and block 40 4098. A phoneme block is associated with a unique canonical phoneme and each phoneme block provides a prediction for how likely it is that the time block contains an audio segment that represents the phoneme with which the phoneme block is associated. The likelihoods of the predictions of the phoneme blocks combined sum to 1.0. The likelihood predictions of all forty phoneme blocks are collectively given to the argmax function 4100, which then returns the canonical phoneme associated with the phoneme block that produced the highest likelihood. The canonical phoneme that returns from the argmax functionality 4010 is the decoded phoneme 4012. Since the audio data blocks individually can have very short lengths, for example 10 milliseconds, the phoneme-change functionality 4014 checks to see if the decoded phoneme 4012 of the current audio data block matches the decoded phoneme of the previous audio data block. If that is the case, then the decoded phoneme 4012 of the current audio data block is discarded. The reason for this, is that a spoken sequence that in reality only represents a single phoneme can, for example, have a length that is longer than multiple time blocks combined. For example, saying “Alexa” out loud takes about 1 sec, which would be split into 100 audio data blocks, however, only five individual canonical phonemes, assuming a perfect pronunciation, are represented. Such a case results in multiple audio data blocks representing the same phoneme which, if not filtered, would increase the difficulty of detecting a keyword variant. The output of the trained acoustic model 2008 is the predicted sequence of phonemes 2014.

[0064] Fig. 3c shows an example 5000 of a keyword recognition apparatus 2000 in action. Each line 5002 of the example is an output to a terminal on a computer. Each line, where each line represents an audio data block 4016, states the timestamp 5004 of the audio data that is being processed by the trained acoustic model. The output comprises a plurality of predictions 4099, wherein a value lies between 0.0 and 1.0 for each possible phoneme 5006, whereby the value represents the predicted likelihood.

[0065] Fig. 3d shows a table 6000 depicting a label 6002 for each phoneme 6004, whereby the label is a number between 0 and 40. The labels “silence” 6006 and “blank” 6008 stand for when no phoneme is registered in the audio data block 4016. In Fig. 3c, when a phoneme is registered 5006, the label associated with the phoneme 5008 and the phoneme 5010 are returned. Once a predicted sequence of phonemes 2014 matches a keyword 2022, the detected keyword and the timestamp are returned 5016.

[0066] Fig. 4a shows an overview of the functionality of a media processor 102. The media processor 102 comprises a microphone 102c, a keyword recognition apparatus 2000 comprising an input interface 102d and an output interface 2024, and a processor 102b comprising a network and input interface 3006. The overview of the functionality of a media processor 102 further comprises a first media provider 3010 and a second media provider 3014. The microphone 102c is connected to the input interface 102d, the input interface 102d is connected to the output interface 2024, the output interface 2024 is connected to the network / input interface 3006, and the network / input interface 3006 is connected to both the first media provider 3010 and to the second media provider 3014. Subsequently, the functionality of the media processor 102 is explained. The media processor 102 combines the keyword recognition apparatus 2000 with additional components to form a fully functioning media system with an input 2002 and an output 3008, 3012. The microphone 102c is continuously recording its surroundings to create an input audio data 2002 consisting of a stream of input audio data, which it continuously feeds to the keyword recognition apparatus 2000. When the keyword recognition apparatus 2000 detects a keyword 1002, it will send an action 2026 to the processor 102b. The action 2006 can, for example, tell the media processor 102 to change into a sleep mode or to change into a power mode. It is also possible for other actions to be sent out via a network / input interface 3006 in the form of messages 3008, 3012 to other media providers. In the embodiments depicted in Fig. 4a, the media processor has a capability to send a message to either a first media provider 3010 or a second media provider 3014, however, in other embodiments, a media processor 102 can have the capability to send messages to a higher number of media providers.

[0067] Fig. 4b gives an overview 4500 of the steps describing an example embodiment for how the keyword recognition apparatus 2000 and the keyword variant generator 1000 work together. The overview 4500 comprises of three columns: offline 4600, keyword configuration 4700 and online 4800, wherein offline 4600 comprises the block 1 4602, with the description: “Train acoustic model or use pre-trained model”, and block 2 4604, with the description: “Compute phoneme statistics for the given model - this may include handcrafting or in the extreme case be fully handcrafted”. The column keyword configuration 4700 comprises block 3 4702, with the description: “For each keyword of interest, generate a list of phonetic variants based on its canonical phoneme sequence using the statistics - one variant being one phoneme sequence”. The column online comprises of block 4 4802 with the description: “Perform interference with the acoustic model”, of block 5 4804, with the description: “Perform simple decoding on the prediction, such as argmax decoding”, of block 6 4804 with the description: “Comparison: Match the decoded phoneme sequence against the stored list of phonetic variants”, and of block 74806 with the description: “Issue keyword detection and reset in case of match”. Block 1 4602 is connected to block 2 4604, block 2 4604 is connected to block 3 4702, block 44702 is connected to block 54802, block 34702 and block 54802 are connected to block 64804, and block 64804 is connected to block 74806. Subsequently, the functionality is explained. The blocks in the column offline 4600 describe processes that are performed to set-up a keyword recognition apparatus 2000 before it becomes available to a user. The blocks in the column online 4800 describe processes that a keyword recognition apparatus 2000 undertakes while it is in use. The column keyword configuration 4700 describes processes that can be performed online in an embodiment or offline in a different, since it is possible to tune a keyword recognition apparatus 2000 based on a test set of keyword variants, which is performed offline, and the generation of keyword variants as performed by a keyword variant generator 1000, as described in Fig. 1 , is performed online. Due to the separation of the keyword recognition apparatus 2000 and the keyword variant generator 1000, an embodiment with an online keyword configuration 4700 does not require a trained acoustic model 2008 to be re-trained for a new keyword, meaning that in such an embodiment, the trained acoustic model 2008 can be very generic.

[0068] The use of an embodiment of a keyword recognition apparatus 2000 in accordance with the present invention is not applicable to general automatic speech recognition, as automatic speech recognition is about transcribing text without prior expectation, whereas key word detection is about detecting something given. In other words, automatic speech recognition is trying to understand everything a person says, whereas a keyword recognition apparatus 2000 is only listening out for keywords.

[0069] Further embodiments comprise, in a non-exhaustive list, a trained acoustic model 2008 that uses CTC loss, and / or a trained acoustic model 2008 that uses RNN-T architecture, and / or a trained acoustic model 2008 that uses a non-peaky, i.e. densely labelled, loss function.

[0070] An example for the generation of variant keyword sequences of phonemes can be seen in the following, whereby the number of keyword variants has been restricted to three. The keyword 1002 “Alexa” is input to the keyword variant generator 1000, where “Alexa” is translated into its canonical phoneme sequence, e.g. using a phonemizer: IQI / I / / £ / / K / / S / / Q / . Variants are then generated by substituting phonemes based on phoneme confusion statistics, for example: IQI / I / / £ / / K / / S / IQI, IQI / I / / £ / / K / / Z / IQI, IQI / I / / £ / / g / / s / IQI. The confusion statistics are constructed by analysing model-specific statistics, e.g. distances between phoneme representations in a model’s latent space. Then, a set of phoneme sequences, in this example three sequences representing possible pronunciations of “Alexa”, is returned. While this is a very simple example, it highlights the idea behind the process of the keyword variant generator 1000.

[0071] Fig. 7a illustrates a user device 100 comprising a device hardware 102. The device hardware comprises a storage 102c for detection information, wherein the storage 102c comprises a plurality of storage entries 103a, 103b, 103c, where each storage entry has a specific detection information item associated with a data instance identification of a specific data instance of a plurality of different data instances 401 , 402, 403, 404, 405. The storage 102c is illustrated, in more detail, in Fig. 7b showing the individual entries 103a, 103b, 103c. Each entry has a certain detection information for a specific data instance which is, for example, a store, and the corresponding identification is in the first column of the schematic illustration in Fig. 7b. Hence, the store or data instance with the identification ID2 has a certain detection information 2 which is a wake word 2, i.e., for this store. In the example, the second data instance is an electronic devices store and, therefore, the wake word would be, for example, the name of this store while wake word 1 in entry 103a would be the wake word of the supermarket under consideration that operates as the data instance with ID1. In an embodiment, that there can be multiple wake words configured for a single service, so that a detection of not only one wake word but e.g. two wake words results in setting up a connection to one and the same data instance or store.

[0072] The hardware comprises a monitoring device 102a which can be a microphone or a camera for acoustically detecting or visually detecting the detection information. In case of a visual detection scenario, the detection information would be a certain picture or a certain gesture made by a human. The device hardware comprises a processor 102b that is implemented to, in an embodiment, not only control the whole device hardware but to also perform a certain detection, but typically under the control of the media runtime device software 110. This software 110 comprises a media interface 114 to one or more data instances 401 to 405 and this media runtime device software is configured for processing a monitoring result of the monitoring device 102a to obtain a detected detection information item of a plurality of different detection information items. The runtime device software is configured for accessing the storage to obtain the data instance identification and to finally initiate a connection to the specific data instance identified by the certain detection information, i.e. , by the certain wake word using the data instance identification of the specific data instance.

[0073] Hence, for this purpose, the user device may only send the identification information of the certain data instance or the detected wake word or both information in order to notify the receiver of this message that a connection to the specific data instance is to be established. Hence, the sending out of such an information is a minimum implementation of the initiating a connection to the specific data instance. Further steps until this data connection if finally established can be done either by the user device or by a cloud backend service or by any other instance.

[0074] Subsequently, Fig. 7c is illustrated to show a procedure mainly performed by the user device 100 of Fig. 7a. In a step 102, a monitoring of the environment using the monitoring device 102a such as the microphone or the camera is performed. This monitoring can be a continuous monitoring or only a monitoring at certain time periods or a monitoring that is activated by a certain trigger such as a detection of an increase of acoustic or optical energy around the user device. The trigger to start monitoring can also come through the HAL (e.g. if the device has a button to enable / disable listening).

[0075] In step 804, the environment recording output by block 802 is processed in order to detect one of a plurality of predetermined detection information such as to find out whether a certain key word of a plurality of key words is included in the environment recording. Subsequent to the detection of a certain detection information item, step 806 performs a looking up of data instance information in the storage 102c in the user device in order to obtain the specific data instance that is associated with the certain wake word.

[0076] In step 808, the connection to the specific data instance is initiated. This initiating can be the triggering of a connection to the selected data instance either directly or via a cloud back-end service or via any other means. As soon as the connection is established, the user device or a user application performs certain transactions to or from the selected data instance directly or indirectly as illustrated in step 810 in order to, for example, browse the content of a data instance and selecting a certain item from the content for the purpose of buying this item or for any other procedure that can be done via a user profile on a data instance. Another such application can be, for example, the entering of data in the web front end of, for example, an insurance company as a data instance or similar procedures. All these back-and-forth communications in order to perform such transactions are collectively included in step 810 of Fig. 7c.

[0077] The cloud-supported embodiment is specifically suited for updating operations that are, for example, necessary for data instances, when these data instances, for example, change their name. Naturally, changing the name of a store, for example, should also be migrated to all the individual user devices. However, this is all but easy since a supermarket provider, for example, does not have immediate access to all user devices that have an account at this data instance. A preferred procedure performed to a wake word change is illustrated in Fig. 8

[0078] When the name of a supermarket, for example, is changed, or when one supermarket takes over the other supermarket, the taking over supermarket is interested in also changing the wake word in order to make sure that the supermarket is called by its name rather than its (earlier) competitor’s name. To this end, the data instance in question, such as data instance 400, changes its specific detection information such as its wake word which is illustrated in step 820. This step 820 takes place within the data instance. In step 822, the data instance accesses the cloud back end and, particularly, this accessing takes place via the communication interface 204 of the cloud backend.

[0079] In step 824, a typical authorization step takes place so that the accessing data instance authorizes itself with respect to the cloud backend. In step 826, the cloud backend receives the new wake word from the data instance that has been correctly authorized and then stores the new detection information for this data instance and, preferably, also deletes the old detection information. This takes place in the cloud backend library 208 illustrated in Fig. 9.

[0080] In step 828, the cloud backend activates an update service for the specific data instance that has changed its wake word. This activation results in a generation of a certain information and / or message within the cloud backend that this specific data instance has changed the wake word. One result of this activation is that a login of a user device is specifically detected in step 830. Alternatively, the wake word change independent automatic detection of a login of a user device can also be specifically used by the cloud backend update service.

[0081] In step 832, the user device is asked for the data instance list, i.e., for the individual data instances that have stored, within the user device, a certain wake word. Therefore, generally, the cloud backend asks the user device for basically the information in the first column of the storage content description of Fig. 7b. In step 834, the cloud backend receives this list subsequent to the user device having established this list and having sent this list to the cloud backend. Step 834 illustrates that the update service is initiated as soon as the specific data instance that has changed its wake word is located on the list provided by the user. The new wake word is communicated to the user device and, in step 836 performed by the user device, the old detection information in the storage is replaced by the new detection information in the specific storage place for this specific data instance.

[0082] Fig. 6a and 6b illustrate a procedure showing a sequence of steps / message between the user 99, the user application 300, the cloud backend 200, the media runtime device software 110 and the content service 400 for playing content with the cloud ecosystem. The playing of content or the uploading of content within the ecosystems involves actions of the corresponding participants of the ecosystem such as device makers, content services, users. It is to be noted that the heavy-weight data or content is retrieved directly by the device media runtime software of the user device, while the cloud backend only handles the playback commands. The data of authorized devices and authorized content services for a user are stored and retrieved from databases that are part of the cloud backend. These databases contain all data that is necessary for authorized access to a content service, where tokens are needed for access. When, in the Fig. 6a, Fig. 6b embodiment such data is looked-up in the cloud backend, it is retrieved from a corresponding database 208.

[0083] It is to be noted that the illustration in Fig. 6a makes clear that each message, such as a message 700 between the user application 300 and the cloud backend 200 refers to two actions of the entities involved. The first action is that the user application sends out the message 700 via the cloud backend interface 304, and the second action performed by the cloud backend 200 is that the cloud backend 200 receives the message 700 via the application interface 206. For each of the corresponding messages that are exchanged between two entities in the Fig. 6a, Fig. 6b illustration, there exist the two actions, i.e., on behalf of the entity sending the message and on behalf of the entity receiving the message, where the corresponding interfaces involved are clear from the illustration. Hence, although not explicitly outlined all the time, these two actions are always performed. Subsequently, Fig. 6a and Fig. 6b are discussed in more detail. In the action 602, the user 99 opens the user application running on the user application apparatus. In step 603, the user logs into the app and, in reply to this logging into the app, the user authenticates herself or himself via the message 700 from the user application to the cloud backend. In order to further use the cloud backend, the user receives a token including the user’s permissions and the corresponding user ID in message 605. The user token is used in all calls to the cloud backend to identify the user and to check if access is allowed for the requested function for the user.

[0084] In order to finally come to a playing of a media content, the user application sends a request for a list of content services illustrated at 702. All content services that are authorized in the cloud backend are returned. Each content service is either authorized or not authorized by the user. Hence, due to the fact that the cloud backend replies to this message 702, with a full selection of content services authorized in the ecosystem, the selection what is valid for the specific user is performed in step 704. Here, the cloud backend checks, which content services are actually authorized for the user.

[0085] For each identified content service, it is checked, if the user is allowed to use the content service. If the user is not allowed, the content service is removed from the result. In a step 708, it is checked, whether the content service allows the user. Probably, the user is denied for some reason, for example by not having paid a bill, etc. This is done in step 708. In step 710, the cloud backend returns a list of allowed content services to the user application 300. This list is presented to the user via the user interface in the user application.

[0086] In step 717a, the user device and specifically the media device runtime software 110 detects a wake word of a plurality of stored wake words. In step 717b, a look up to the storage in the user device is performed, and the data instance identification information is retrieved and used for forming a message in step 717c.

[0087] The user device sends, subsequent to the detection of e.g., a wake word, the message 715 to the cloud backend for activating a specific user instance which can be a store, such as a supermarket, a shop for electronic devices or a supplier of media data, such as streaming data or any other material or non-material goods or services. In response to that, the cloud backend looks up, whether credentials for this content services are stored in the cloud backend. This is done in step 716, and when such user credentials for the requested content service are found, an authentication message 718 is sent from the cloud backend to the content service via the communication interface 204 of the cloud backend. The result message 720 is illustrated in Fig. 6a and the result message can, for example, comprise a certain token, from the content service, or any other positive authentication result. In response to the positive result 720, the cloud backend requests content metadata in a message 722 from the content service 400, and the content service 400 sends back the requested content metadata in message 724. This content metadata includes an information in the detected data instance belonging to the detected wake word. In the embodiment illustrated in Fig. 6a, a message 725 comprising this data instance information is sent to the application apparatus 300 and displayed to the user 99 via the user interface. In response to a positive confirmation message 746, a connection activation message 748 is sent to the cloud backend 200. In order to be able to access the data instance under consideration, content metadata such as a playable URL or URI is sent from the cloud backend 200 to the media runtime device software 110 running on the user device 100. Hence, the content metadata are sent from the cloud backend 200 to the media runtime device software 110 on the user device. As stated, this content metadata can, for example, be the playable URL or any other data, by which the media runtime device software 110 running on the user device can request the content via a message 752 from the user device media interface 114. As soon as the playable URL, for example, is received by the remote data instance, the remote data instance streams back the content as shown in message 756 and this takes place via the media interface 114 of the user device 100 and the content is started to be replayed as illustrated at 754. For replaying the content, the content is downloaded 756 from the data instance. It is a specific feature of the embodiment, that the content received in step 756 is already a content specific for the user at the specific data instance selected by the wake word detected in step 717. Such a content specific for the user can be the products bought by the user in earlier sessions of suggestions for the specific user made by the data instance or any other user specific data instance interface personalized by the user under consideration. The user ID is forwarded by the cloud backend to the content service in the message 752, but can also be forwarded earlier or at any other stage in the processing, i.e., in message 718 or 722.

[0088] In step 756, the user-specific content such as the user- personalized data instance interface is rendered by the user device. This rendering can take place on the user device itself via a specific interface 770 such as a touch sensitive interface on the user device. The (human) user can input a selection of a certain item via the interface 770 and this selection is forwarded to the data instance in step 772, and the data instance can react in step 774. Generally, a dialogue can take place between the user 99 and the data instance 400 via the interface 770.

[0089] As illustrated in Fig. 6b, other alternatives for this communication can be implemented as well. One alternative is that the content received in step 756 is forwarded to the cloud backend 200 in step 758 and from there, via message 760, to the user application running on device 300 provided that the device 300 is (physically or logically) separated from the user device. The user reaction 762 such as a selection of a specific item from the user-personalized interface from the data instance 400 is received and the selection is forwarded 764 to the cloud backend 200 and from there via message 766 to the media runtime device software 110. This instance sends the user-selection retrieved from message 766 to the data instance in step 772 using message protocol information such as a playable URL received from the cloud backend 200 in message 766, and a corresponding reply is received in step 774. Other communication protocols apart from using playable URLs can be implemented as the case may be.

[0090] An embodiment is illustrated in Fig. 5 illustrating several elements of the invention. The hatched line indicates what is included in the typically mobile device owned by the user. The Media CDN (content delivery network) is located anywhere in the cloud, i.e., remote from the user device 10 or remote from the apparatus in which the application program is executed, by which the user can control her or his user devices 10. In this embodiment, the mobile device has an interface for interfacing with the Media CDN in the cloud.

[0091] Fig. 5 shows an example of an ecosystem in particular in the case of providing media content (the same may be obtained, however, for generic services to be provided, and not only media streams to be transmitted). Fig. 5 shows a player device 10. Each device 10 may control a device hardware 1 (which may be a display device and / or loudspeakers and / or a hardware connection to the loudspeakers. The device 10 may receive a service e.g. media content 51 (which may be divided between 51a and 51 b in some examples) from a media content distribution network (CDN) 5 (which may be divided between media update CDN 5a, sending 51a, and CDN 5b, sending 51b, in some examples, but this is not strictly necessary; 51a, 51b, 51 are examples of 202). The assisting system (which may be a cloud backend system) 4 may control the player device 10 (10a, 10b, 10c, 10d), e.g., through content control 38 and / or playback control 34. The player device 10 may include a device SDK / player 1.

[0092] The assisting system 4 may control the playback and the reception of the media streams through control commands 34, 48, received through the first remote connection 401. The media CDN 5 (second processing system in Figs. 6a and 6b) which provides the content is not the same as the cloud backend system (assisting system) 4. The content 51 (51a, 51b, 202) is transported through the third remote connection 403 and not the first remote connection 401 as the control commands 34 and 38 (it may be that they both share the same physical connection, but anyway the content 51 and the control commands 34 and 38 are not in the same session). The device SDK / player 2 may provide the media content (e.g., audio and / or video streams) to a content decryption service 28, which can therefore decrypt the media content 27 (combing from 51 , 51a, 51b, 202). The decrypted media content 29a may therefore be provided to a hardware media Tenderer unit 1. The hardware media Tenderer unit 1 may include the hardware abstraction layer (HAL) 11. The HAL 11 may be provided by an operating system. The HAL 11 may control the rendering of the audio and / or video. The HAL 11 may be controlled by hardware control and / or hardware events 26, e.g. sent by the device SDK / player 2. The device SDK / player 2 may provide the hardware control and hardware events 26 based, in particular, on the control 34 and 38 accepted by the cloud backend system 4 (assisting system). The player device 10 may be connected to a local network (e.g., LAN) 220. The device SDK / player 2 may include a web browser 21. The device SDK / player 2 may include a media player 22. The device SDK / player may contain a cache 23 (e.g. in the media player 22). The device SDK / player 2 may contain a group playback entity 24.

[0093] It is to be mentioned here that all alternatives or aspects as discussed before can be used individually, i.e., without any other alternative or object than the contemplated alternative, or object. However, in other embodiments, two or more of the alternatives or the aspects can be combined with each other and, in other embodiments, all aspects, or all alternatives can be combined to each other.

[0094] Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Analogously, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus.

[0095] Depending on certain implementation requirements, embodiments of the invention can be implemented in hardware or in software. The implementation can be performed using a digital storage medium, for example a floppy disk, a DVD, a CD, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed. Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed. Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may for example be stored on a machine-readable carrier. Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine-readable carrier or a non-transitory storage medium. In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer. A further embodiment of the inventive methods is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein. A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may for example be configured to be transferred via a data communication connection, for example via the Internet. A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein. A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein. In some embodiments, a programmable logic device (for example a field programmable gate array) may be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware apparatus.

[0096] The above-described embodiments are merely illustrative for the principles of the present invention. It is understood that modifications and variations of the arrangements and the details described herein will be apparent to others skilled in the art. It is the intent, therefore, to be limited only by the scope of the impending patent claims and not by the specific details presented by way of description and explanation of the embodiments herein.

Claims

Claims1. Keyword recognition apparatus (2000), comprising: an input interface (102d) for receiving an input audio data (2002); a trained acoustic model (2008) configured for processing the input audio data (2002) and for outputting a predicted sequence of phonemes (2014); a memory (102c) for storing a plurality of variant keyword sequences of phonemes (1018) for a specified keyword; a keyword matcher (2016) for detecting a match between the predicted sequence of phonemes (2014) and a variant keyword sequence of phonemes (1014) of a plurality of variant keyword sequences of phonemes (2020); and an output interface (2024) for performing an action (2026) in response to a detection of the match (2022).

2. A keyword recognition apparatus (2000) according to claim 1 , wherein the memory (102c) is configured to store a second plurality of variant keyword sequences of phonemes for a second specified keyword, different from the specified keyword, wherein the keyword matcher (2016) is configured to compare the predicted sequence of phonemes (2014) to the plurality of variant keyword sequences of phonemes (2020) and the second plurality of variant keyword sequences of phonemes (2021), and to detect the match for a matching specified keyword, the matching specified keyword having a variant keyword sequence of phonemes (1014) of its associated first (2020) or second plurality of variant keyword sequence of phonemes (2021) being in a predefined relation to the predicted sequence of phonemes (2014).

3. A keyword recognition apparatus (2000) according to claim 2, wherein the output interface (2024) is configured to perform a second action for the second specifiedkeyword, the second action being different from the first action for the specified keyword.

4. A keyword recognition apparatus (2000) according to claim 3, wherein the output interface (2024) is configured to perform either the first (2026) or the second action depending on whether the matching specified keyword is the specified keyword or the second specified keyword.

5. A keyword recognition apparatus (2000) according to any one of claims 3 or 4, wherein the output interface (2024) is configured to determine: the action (2026) in case a match for the specified keyword is detected, or the second action in case a match for the second keyword is detected.

6. A keyword recognition apparatus (2000) according to any one of claims 2-5, wherein the keyword matcher (2016) is configured to compare the predicted sequence of phonemes (2014) to the plurality of variant keyword sequences of phonemes (2020) in parallel to comparing the predicted sequence of phonemes (2014) to the second plurality of variant keyword sequences of phonemes (2021).

7. A keyword recognition apparatus (2000) according to any one of claims 1-6, wherein the keyword matcher (2016) is configured to perform the comparison using a separate state machine (4002, 4004, 4006, 4008) for each variant keyword sequence of phonemes (1014) for the comparison of the predicted sequence of phonemes (2014) to the plurality of variant keyword sequences of phonemes (2020) and to the second plurality of variant keyword sequences of phonemes (2021), or to perform the comparison for a second plurality of variant keyword sequences of phonemes (2021) with a separate state machine (4002, 4004, 4006, 4008).

8. A keyword recognition apparatus (2000) according to claim 7, wherein each state machine has associated therewith an indication of a specific wake word with which the corresponding variant keyword sequence of phonemes (1014) is associated, andwherein the keyword matcher is configured to identify either the specified key word or the second specified keyword using the indication of the specific wake word associated with the state machine that has detected a match in the comparison of the predicted sequence of phonemes (2014) to its associated variant keyword sequence of phonemes.

9. A keyword recognition apparatus (2000) according to any one of the claims 1 - 8, wherein the trained acoustic model (2008) comprises: a trained neural network block (2010) for receiving the input audio data (2002) and for outputting a probability of predictions (4099), and a prediction decoder (2012) for decoding the plurality of predictions (4099) to obtain the predicted sequence of phonemes (2014).

10. A keyword recognition apparatus (2000) according to any one of claims 1-9, wherein the input interface (102d) is configured to receive, as the input audio data (2002), an audio file, or wherein the input interface (102d) is configured to receive a stream of audio data sequences, and wherein the input interface (102d) is configured to restructure an input audio data (2002) or a stream of audio data sequences into a sequence of blocks of a certain length.

11. A keyword recognition apparatus (2000) according to any one of claims 1-10, wherein the input audio data (2002) is structured to comprise of a plurality of audio data blocks (4015), each with a certain length, wherein the certain length is selected from a range of lengths between 1 ms and 1s, wherein the trained neural network block (2010) is configured to make a plurality of predictions (4099) on every audio data block or on only every second audio data block or on only a nthdata block, wherein n is larger than 2 and smaller than 10, andwherein each prediction of the plurality of predictions (4099) is associated with a different phoneme.

12. A keyword recognition apparatus (2000) according to claim 11 , wherein a prediction of the plurality of predictions (4099) comprises a likelihood, and the likelihoods of the plurality of predictions (4099) sum to 1.

13. A keyword recognition apparatus (2000) according to claim 11 or 12, wherein the prediction decoder (2012) is configured to perform an argmax decoding (4100) among the plurality of predictions (4099) to obtain a sequence of results when generating the predicted sequence of phonemes.

14. A keyword recognition apparatus (2000) according to claim 13, wherein the prediction decoder (2012) is configured to perform a processing of the sequence of results in order to generate the sequence of phonemes.

15. A keyword recognition apparatus (2000) according to claim 14, wherein the prediction decoder is configured to perform, as the processing, a deletion of repetitions of phonemes in the sequence of results or a deletion of silence portions, or filtering out phonemes in the sequence of results if the predicted phoneme (4012) comprises of the same phoneme as the predicted phoneme (4012) of the previous audio data block for which a plurality of predictions (4099) was made.

16. A keyword recognition apparatus (2000) according to any one of claims 1-15, wherein the memory (102c) is configured to store a plurality of three or more keyword sequences of phoneme variants (1014) for each keyword, or wherein the memory (102c) is configured to store a number of groups of variant keyword sequences of phonemes for a number of different specified keywords, wherein the number of specified keywords being at least five.

17. A keyword recognition apparatus (2000) according to any one of claims 1-16, further comprising a voice activity decoder (2004), wherein the trained acoustic model (2008) is configured to receive a signal (2003) from the voice activity decoder (2004), whereinthe trained acoustic model (2008) is configured to change its behaviour in accordance to the received signal (2003), or to start generating an output only on reception of the signal (2003) from the voice activity decoder (2004) indicating a voice activity in the input audio data (2002).

18. A keyword variant generation apparatus (1000), comprising: an input interface (1004) for receiving a keyword; an analyser (1008) for analysing the keyword to obtain an analysed phoneme sequence (1010); a keyword variant producer (1012) for producing a plurality of variant keyword sequences (1014) for the analysed phoneme sequence (1010); and an output interface (1016) for outputting the plurality of variant keyword sequences (1014).

19. A keyword variant generation apparatus (1000) according to claim 18, wherein the analyser (1008) is configured to generate the analysed phonemes sequence (1010).

20. A keyword variant generation apparatus (1000) according to claim 18 or 19, wherein the keyword variant generation apparatus (1000) is configured to use a phoneme confusion statistic.

21. A keyword variant generation apparatus (1000) according to claim 20, wherein the keyword variant generation apparatus (1000) is configured to generate the confusion statistics, or a confusion matrix according to a latent representation of a training data set comprising text-aligned speech samples comprising audio input and corresponding sequence of truth labels.

22. Media processor (102), comprising: a keyword recognition apparatus (2000) according to any one of claims 1-17; and a microphone (102c) for capturing an audio signal wherein the media processor (102) is configured to derive the input audio sequence (2002) received by the input interface(102d) from the audio signal; a processor (102b) configured for performing a processing move depending on the action performed by the output interface (2024).

23. A media processor (102) according to claim 22, wherein the processor (102b) is configured to have a sleep mode or power mode, and wherein the microphone (102c) is active in the sleep mode, and wherein the processor (102b) is configured for changing from the sleep mode to the power mode in response to the action performed by the output interface (2024).

24. A media processor (102) according to claims 22 and 23, wherein the memory (102c) is configured to store a second plurality of variant keyword sequences of phonemes (2021) for a second specified keyword, different from the specified keyword, wherein the keyword matcher (2016) is configured to compare the predicted sequence of phonemes (2014) to the plurality of variant keyword sequences of phonemes (2020) and the second plurality of variant keyword sequences of phonemes (2021), and to detect the match for a matching specified keyword, the matching specified keyword having a variant keyword sequence of phonemes of its associated first (2020) or second plurality of variant keyword sequence of phonemes (2021) being in a predefined relation to the predicted sequence of phonemes (2014); wherein the output interface (2024) is configured to perform a second action for the second specified keyword, the second action being different from the first action (2026) for the specified keyword; and wherein the processor is configured to generate a first message (3008) to a first external data instance (3010) in response to the action (2026) and a second message (3012) for the second different external data instance (3014) in regard to the second action.

25. A media processor (102) according to any one of claims 22-24, comprising:A keyword variant generating apparatus (1000) according to claim 15-18, wherein the output interface (1016) of the keyword variant generation apparatus is connected to the memory (102c) of the keyword recognition apparatus (2000).

26. Method of operating a keyword recognition apparatus (2000) according to claim 1 , comprising: receiving, to an input interface (102d), an input audio data (2002); configuring a trained acoustic model (2008) for processing the input audio data (2002) and for outputting a predicted sequence of phonemes (2014); storing, on a memory (102c), a plurality of variant keyword sequences of phonemes (2020) for a specified keyword; detecting, via a keyword matcher (2016), a match between the predicted sequence of phonemes (2014) and a variant keyword sequence of phonemes of a plurality of variant keyword sequences of phonemes (2020); and performing, via an output interface (2024), an action (2026) in response to a detection of the match.

27. Method of operating a keyword variant generation apparatus (1000) according to claim 18, comprising: receiving, via an input interface (1004), a keyword; analysing, via an analyser (1008), the keyword to obtain an analysed phoneme sequence (1010); producing, via a keyword variant producer (1012), a plurality of variant keyword sequences (1014) for the analysed phoneme sequence (1010); and outputting, via an output interface (1016), the plurality of variant keyword sequences (1014).

28. Method of operating a media processor (102) according to claim 22, comprising: operating a keyword recognition apparatus (2000) according to claim 26; and capturing, via a microphone (102c), an audio signal wherein the media processor (102) is configured to derive the input audio sequence (2002) received by the input interface from the audio signal; a processor (102b) configured for performing a processing move depending on the action performed by the output interface.

29. Computer program for carrying out any one of the methods of claims 26-28.

Images