Encoding method, decoding method, program, encoding device, and decoding device

Abstract

This encoding method comprises: encoding an input signal to generate first encoded data and a plurality of pieces of second encoded data having lower occurrence frequency than the first encoded data, the pieces of second encoded data having different occurrence frequencies; and multiplexing the first encoded data, the plurality of pieces of second encoded data, and identification information into a bit stream. The identification information indicates, for each of a plurality of pieces of encoded data including the first encoded data and the plurality of pieces of second encoded data, the presence or absence of the second encoded data having lower occurrence frequency than the encoded data.

Description

Encoding method, decoding method, program, encoding device, and decoding device

[0001] The present disclosure relates to an encoding method, a decoding method, a program, an encoding device, and a decoding device.

[0002] In recent years, artificial intelligence (AI) has emerged, and AI that solves various problems has been developed. For example, data detected by a sensor is transmitted to a monitoring terminal, and the monitoring terminal performs real-time AI analysis on the received data, detects the occurrence of a problem, and takes countermeasures.

[0003] Patent Literature 1 discloses a technique for multiplexing time information corresponding to detection information when multiplexing and transmitting detection information acquired by a plurality of sensors with different timings. This enables the receiving side to use the time information to synchronize the acquisition timings between pieces of detection information for use.

[0004] Japanese Unexamined Patent Publication No. 2015-93123

[0005] The present disclosure provides an encoding method, a decoding method, and the like that can reduce the amount of information required for identification information for identifying information included in a bitstream.

[0006] One aspect of the encoding method according to the present disclosure includes encoding an input signal to generate first encoded data and a plurality of second encoded data that have lower occurrence frequency than the first encoded data and have mutually different occurrence frequencies, multiplexing the first encoded data, the plurality of second encoded data, and identification information into a bitstream, wherein the identification information indicates, for each of a plurality of encoded data including the first encoded data and the plurality of second encoded data, whether there exists a second encoded data having a lower occurrence frequency than the encoded data.

[0007] Furthermore, one embodiment of the decoding method according to the present disclosure decodes a bitstream to separate a first encoded data, a plurality of second encoded data having a lower occurrence frequency than the first encoded data and having different occurrence frequencies from one another, and identification information, wherein the identification information indicates whether or not there is a second encoded data having a lower occurrence frequency than the encoded data for each of the plurality of encoded data including the first encoded data and the plurality of second encoded data.

[0008] Furthermore, this disclosure can be implemented not only as the above-mentioned encoding method or decoding method, but also as a program that causes a computer to execute the above-mentioned encoding method or decoding method. Moreover, it can also be implemented as a computer-readable recording medium that stores the program.

[0009] The encoding method, decoding method, program, encoding device, and decoding device relating to this disclosure can reduce the amount of information required for identification information to identify information contained in a bitstream.

[0010] Figure 1 is a diagram showing an example of the application of an information processing system according to an embodiment. Figure 2 is a diagram showing an example of the configuration of a conventional bitstream. Figure 3A is a diagram showing an example of the configuration of a bitstream according to an embodiment. Figure 3B is a diagram showing another example of the configuration of a bitstream according to an embodiment. Figure 4 is a block diagram showing an example of the implementation of a transmitting terminal according to the first embodiment. Figure 5 is a block diagram showing an example of the configuration of a transmitting terminal according to the first embodiment. Figure 6 is a flowchart showing an example of the operation of a transmitting terminal according to the first embodiment. Figure 7 is a block diagram showing an example of the implementation of a monitoring terminal according to the first embodiment. Figure 8 is a block diagram showing the functional configuration of a monitoring terminal according to the first embodiment. Figure 9A is a flowchart showing an example of the operation of a monitoring terminal according to the first embodiment. Figure 9B is a flowchart showing another example of the operation of a monitoring terminal according to the first embodiment. Figure 10 is a diagram showing an example of a pattern table according to the second embodiment. Figure 11A is a diagram showing an example of the configuration of a bitstream including pattern information according to the second embodiment. Figure 11B is a diagram showing another example of the configuration of a bitstream including pattern information according to the second embodiment. Figure 12A is a flowchart showing an example of the operation of a monitoring terminal according to the second embodiment. Figure 12B is a flowchart showing another example of the operation of a monitoring terminal according to the second embodiment. Figure 13A is a diagram showing an example of the configuration of a bitstream including an exception code according to the third embodiment. Figure 13B is a diagram showing another example of the configuration of a bitstream including an exception code according to the third embodiment. Figure 14A is a flowchart showing an example of the operation of a monitoring terminal according to the third embodiment. Figure 14B is a flowchart showing another example of the operation of a monitoring terminal according to the third embodiment. Figure 15A is a diagram showing yet another example of the configuration of a bitstream including an exception code according to the third embodiment. Figure 15B is a diagram showing yet another example of the configuration of a bitstream including an exception code according to the third embodiment. Figure 16A is a flowchart showing yet another example of the operation of a monitoring terminal according to the third embodiment. Figure 16B is a flowchart showing yet another example of the operation of a monitoring terminal according to the third embodiment. Figure 17A is a diagram showing an example of the configuration of a bitstream including supplementary information according to the fourth embodiment.Figure 17B is a diagram showing another example of the configuration of a bitstream including complementary information according to the fourth embodiment. Figure 18A is a flowchart showing an example of the operation of a monitoring terminal according to the fourth embodiment. Figure 18B is a flowchart showing another example of the operation of a monitoring terminal according to the fourth embodiment. Figure 19A is a flowchart showing yet another example of the operation of a monitoring terminal according to the fourth embodiment. Figure 19B is a flowchart showing yet another example of the operation of a monitoring terminal according to the fourth embodiment.

[0011] (Knowledge forming the basis of this disclosure) With the rise of artificial intelligence (AI) in recent years, the development of AI to solve various problems is progressing. Data acquired from advanced sensors is sent to a monitoring terminal that includes an AI-based judgment unit, and the monitoring terminal performs real-time AI analysis on the received data to detect the occurrence of problems and take countermeasures. There are many examples of applications for such technology.

[0012] For example, it can be applied to a condition monitoring system that monitors the status of numerous industrial robots deployed in a factory. The condition monitoring system senses the operating sounds of the industrial robots and sends them sequentially to a monitoring terminal, which analyzes the received operating sounds in real time. When an abnormality is detected, it notifies the operator or on-site worker, or takes control such as stopping the industrial robot. Other possible applications include monitoring traffic flow on main roads to detect signs of accidents, or observing the health status of patients or the sleeping posture of infants in hospitals to detect abnormal conditions.

[0013] This section will explain using a condition monitoring system for monitoring the status of industrial robots as an example. Figure 1 shows an example of the application of the information processing system 1 according to the embodiment.

[0014] As shown in Figure 1, the information processing system 1 comprises a plurality of transmitting terminals 10 and one monitoring terminal 20. In the information processing system 1, each of the plurality of transmitting terminals 10 is connected to the monitoring terminal 20 via a wireless network 50 so as to be able to communicate with it.

[0015] The transmitting terminal 10 encodes acoustic data (e.g., the sound of the industrial robot's operation) generated by multiple microphones 30 that sense an object OBJ (in this case, an industrial robot) located in space 2, and video data (e.g., video of the industrial robot) generated by multiple cameras 40 into encoded data, and transmits a bitstream of the encoded data multiplexed to the monitoring terminal 20.

[0016] The monitoring terminal 20 receives multiple bitstreams transmitted by each of the multiple transmitting terminals 10. The monitoring terminal 20 generates multiple decoded data by decoding each bitstream. The monitoring terminal 20 performs AI-based analysis using the generated multiple decoded data. As a result, the monitoring terminal 20 can detect industrial robots that are in an abnormal state, or detect industrial robots that are likely to become abnormal in the near future.

[0017] Specifically, the transmitting terminal 10 encodes the main information, which is the audio data, and also encodes the video data as needed, multiplexes these into a bitstream, and transmits them to the monitoring terminal 20.

[0018] The monitoring terminal 20 performs status monitoring using the acoustic decoded data generated by the decoded process. Furthermore, if the bitstream contains video encoded data, it also performs status monitoring using the video decoded data generated by the decoded process.

[0019] In other words, the acoustically encoded data is always included in the bitstream. The frame length of the acoustic encoding process is, for example, 20 ms. Therefore, the bitstream is generated at that time interval.

[0020] In contrast, the time interval at which video encoded data is generated is longer than that of audio encoded data, because shooting is done as needed. In the following explanation, the reciprocal of the time interval is called the time resolution; a shorter time interval means higher time resolution, and a longer time interval means lower time resolution. A short time interval is also referred to as a high frequency of occurrence, and a long time interval as a low frequency of occurrence. Such audio encoded data is an example of first encoded data. Similarly, such video encoded data is an example of second encoded data.

[0021] Furthermore, the degree to which the framing and camera work of the object being filmed affect the judgment of the monitoring terminal differs. Therefore, a priority order is assigned to the video data based on the degree of this influence, and the video data with higher priority has a higher temporal resolution. This improves the judgment performance of the monitoring terminal 20.

[0022] For example, in Figure 1, the priority of the video captured by camera 40 that frames the gripper part of the industrial robot is increased, while the priority of the video captured by camera 40 that frames the joint part of the industrial robot's arm is decreased. This explanation is based on the premise that the gripper part is more prone to failure than the joint part, and different priorities may be set under different premises.

[0023] When constructing a bitstream by multiplexing acoustic and video encoded data with different temporal resolutions in this way, a problem arises in that the amount of information required to identify what kind of video encoded data is contained in that bitstream becomes large. Figure 2 shows an example of a conventional bitstream configuration.

[0024] In the example shown in Figure 2, the bitstream contains three types of video encoded data P1, P2, and P3, and the time resolution of each video encoded data is assumed to be P1 > P2 > P3.

[0025] As shown in Figure 2, in this case there are four types of bitstream configurations. Therefore, 2 bits of information are always required to identify which video encoding data is contained in the bitstream. For example, if the period during which audio data is acquired (i.e., the frame length of the audio encoding process) is 20 ms, the average number of bits required to indicate which video encoding data is contained in the bitstream per second is 2 bits * 50 = 100 bits / second, which presents a problem as it requires a very large amount of information.

[0026] Patent Document 1 is cited as prior art in this field. Patent Document 1 discloses a technique for multiplexing and transmitting detection information acquired by multiple sensors with different timings, while also multiplexing time information corresponding to the detection information. This makes it possible for the receiving side to synchronize the acquisition timing between detection information using the time information.

[0027] The technology disclosed in Patent Document 1 makes it possible to acquire and multiplex detection information at any time, but it requires time information for synchronization, which does not solve the problem of increasing the amount of identification information.

[0028] Therefore, in order to solve these problems, the inventors came up with the following encoding and decoding methods.

[0029] The encoding method of the first embodiment encodes an input signal to generate a first encoded data and a plurality of second encoded data which occur less frequently than the first encoded data and have different occurrence frequencies from one another, multiplexes the first encoded data, the plurality of second encoded data and identification information into a bitstream, and the identification information indicates whether or not there is a second encoded data which occurs less frequently than the encoded data for each of the plurality of encoded data which includes the first encoded data and the plurality of second encoded data.

[0030] This allows identification information to be represented by only one bit indicating whether or not the corresponding encoded data exists, thereby reducing the amount of information required for identification information to identify the information contained in the bitstream.

[0031] The encoding method of the second embodiment is the encoding method of the first embodiment, wherein pattern information, first encoded data, a plurality of second encoded data representing combinations shown in the pattern indicated by the pattern information, and identification information are multiplexed into the bitstream, and the pattern information is information indicating a plurality of patterns, each representing a combination of the plurality of second encoded data multiplexed into the bitstream.

[0032] According to this, it becomes possible to generate bitstreams that support various combinations of video encoding data depending on the pattern, resulting in the creation of highly flexible bitstreams.

[0033] The third embodiment of the encoding method is the encoding method of the first or second embodiment, wherein the identification information includes an exception code, and the exception code indicates, instead of indicating whether or not there is a second encoded data which occurs less frequently than each of the plurality of encoded data, that there is a third encoded data which is different from the second encoded data.

[0034] This allows for the generation of bitstreams containing specific encoded data using exception codes, resulting in the creation of highly flexible bitstreams.

[0035] The encoding method of the fourth embodiment is an encoding method of any of the first to third embodiments, wherein the complementary information, the first encoded data, the plurality of second encoded data, and the identification information are multiplexed into the bitstream, and the complementary information is information indicating whether or not the complementary information of the second encoded data to be multiplexed into the bitstream is performed.

[0036] This makes it possible to generate bitstreams in which encoded data is supplemented by complementary information, resulting in the creation of highly flexible bitstreams.

[0037] The decoding method of the fifth embodiment decodes a bitstream to separate a first encoded data, a plurality of second encoded data having a lower occurrence frequency than the first encoded data and having different occurrence frequencies from one another, and identification information, wherein the identification information indicates whether or not there is a second encoded data having a lower occurrence frequency than the encoded data for each of the plurality of encoded data including the first encoded data and the plurality of second encoded data.

[0038] According to this, identification information can be represented by only one bit indicating whether or not corresponding encoded data exists, thus reducing the amount of information required for identification information to identify information contained in a bitstream.

[0039] The decoding method of the sixth embodiment is the decoding method of the fifth embodiment, wherein the bitstream is decoded to separate pattern information, the first encoded data, the plurality of second encoded data representing combinations shown in the pattern indicated by the pattern information, and the identification information, wherein the pattern information is information representing a plurality of patterns, each representing a combination of the plurality of second encoded data multiplexed in the bitstream.

[0040] According to this, it becomes possible to separate bitstreams that support various combinations of video encoding data depending on the pattern, enabling highly flexible bitstream separation.

[0041] The decoding method of the seventh embodiment is the decoding method of the fifth or sixth embodiment, wherein the identification information includes an exception code, and instead of indicating whether or not there is a second encoded data which occurs less frequently than each of the encoded data, the exception code indicates that there is a third encoded data which is different from the second encoded data.

[0042] This allows for the separation of bitstreams containing specific encoded data using exception codes, enabling highly flexible bitstream separation.

[0043] The decoding method according to the eighth aspect is the decoding method according to any one of the fifth to seventh aspects, wherein the bit stream is decoded to separate complementary information, the first encoded data, the plurality of second encoded data, and identification information, and the complementary information is information indicating whether or not complementation of the second encoded data to be multiplexed into the bit stream is performed.

[0044] This enables separation of a bit stream in which encoded data is complemented by the complementary information, and enables separation of a highly flexible bit stream.

[0045] The program according to the ninth aspect is a program for causing a computer to execute the encoding method according to any one of the first to fourth aspects.

[0046] This enables support for reducing the amount of information required for identification information for identifying information included in the bit stream.

[0047] The program according to the tenth aspect is a program for causing a computer to execute the decoding method according to any one of the fifth to eighth aspects.

[0048] This enables support for reducing the amount of information required for identification information for identifying information included in the bit stream.

[0049] The encoding apparatus according to the eleventh aspect includes a circuit and a memory connected to the circuit, and in operation, the circuit encodes an input signal to generate first encoded data and a plurality of second encoded data that have a lower occurrence frequency than the first encoded data and have mutually different occurrence frequencies, multiplexes the first encoded data, the plurality of second encoded data, and identification information to generate a bit stream, and the identification information indicates, for each of a plurality of encoded data including the first encoded data and the plurality of second encoded data, whether or not there exists second encoded data having a lower occurrence frequency than that encoded data.

[0050] According to this, identification information can be represented by only one bit indicating whether or not corresponding encoded data exists, thus reducing the amount of information required for identification information to identify information contained in a bitstream.

[0051] The decoding device of the twelfth embodiment comprises a circuit and a memory connected to the circuit, wherein the circuit decodes a bitstream in operation to separate a first encoded data, a plurality of second encoded data having a lower occurrence frequency than the first encoded data and having different occurrence frequencies from one another, and identification information, wherein the identification information indicates whether or not there is a second encoded data having a lower occurrence frequency than the encoded data for each of the plurality of encoded data including the first encoded data and the plurality of second encoded data.

[0052] According to this, identification information can be represented by only one bit indicating whether or not corresponding encoded data exists, thus reducing the amount of information required for identification information to identify information contained in a bitstream.

[0053] These comprehensive or specific embodiments may be implemented as a system, method, integrated circuit, computer program, or recording medium such as a computer-readable CD-ROM, or as any combination of a system, method, integrated circuit, computer program, and recording medium. Furthermore, the recording medium may be a non-temporary recording medium.

[0054] (Embodiments) Hereinafter, embodiments of the encoding method, decoding method, program, encoding device, and decoding device relating to this disclosure will be described in detail with reference to the drawings. The embodiments described below are all preferred specific examples of this disclosure. The numerical values, components, arrangement and connection configurations of components, steps, and the order of steps shown in the following embodiments are examples and are not intended to limit this disclosure. Furthermore, among the components in the following embodiments, components not described in an independent claim will be described as any component constituting a preferred configuration.

[0055] Please note that each figure is a schematic diagram and not necessarily a strictly accurate representation. Furthermore, in each figure, substantially identical components are denoted by the same reference numerals, and redundant explanations may be omitted or simplified.

[0056] Furthermore, unless otherwise specified, ordinal numbers such as "first," "second," etc., in this disclosure do not refer to the number or order of components, etc., but are used for the purpose of avoiding confusion and distinguishing similar components, etc.

[0057] [Summary] As described above, in this disclosure, the video encoded data multiplexed into the bitstream is in a hierarchical relationship, and the basis is to reduce the amount of information required for identification information by utilizing this hierarchical relationship.

[0058] Here, a hierarchical relationship refers to a relationship where, assuming there is video encoded data A with high temporal resolution and video encoded data B with low temporal resolution, video encoded data B is generated only when video encoded data A is generated, and the generation of video encoded data A does not depend on the generation of video encoded data B. In this case, video encoded data A is called the "parent video encoded data" of video encoded data B, and video encoded data B is called the "child video encoded data" of video encoded data A.

[0059] For example, the temporal resolution of the video encoding data for video footage captured by a camera framing a part prone to failure is increased, while the temporal resolution of the video encoding data for video footage captured by a camera framing a part less prone to failure is decreased. Then, only when video encoding data with high temporal resolution has been generated can video encoding data with low temporal resolution be generated. In this way, a hierarchical relationship is established between the video encoding data.

[0060] The outline of this disclosure will be explained using Figures 3A and 3B. Figure 3A is a diagram showing an example of the configuration of a bitstream according to an embodiment. Figure 3B is a diagram showing another example of the configuration of a bitstream according to an embodiment.

[0061] In the following, Figure 3A will be used to describe an example in which identification information is placed after the acoustic and video encoded data in the bitstream, and Figure 3B will be used to describe an example in which identification information is placed together at the beginning of the bitstream. In other words, for example, identification information may be placed individually (i.e., in different locations) or together (i.e., in one place).

[0062] The manner in which the identification information is included in the bitstream is not particularly limited in this disclosure. For example, the identification information may be stored together in the bitstream header or metadata. Alternatively, at least a portion of the identification information may be stored in the bitstream header or metadata. Similarly, the following pattern information and complementary information may be placed in the bitstream in a manner not limited to the illustrated examples, and may be placed individually or together, or may be stored in the bitstream header or metadata.

[0063] Here, the identification information and supplementary information correspond to which encoded data, which are shared in advance between the encoding device 100 and the decoding device 200. For example, the identification information and supplementary information may correspond to which encoded data by their placement in the bitstream. Specifically, the identification information may correspond to the encoded data placed before it (e.g., immediately before) and indicate whether or not there is encoded data placed after it (e.g., immediately after). Alternatively, supplementary information may correspond to encoded data that could be placed after it (e.g., immediately after) and indicate whether or not such data is placed. Note that the identification information and supplementary information correspond to which encoded data are not limited to these examples, and may be indicated, for example, by bitstream metadata.

[0064] As shown in Figure 3A, the identification information indicates whether or not child video encoded data exists. In other words, there are two types of identification information. The first type is identification information that indicates that child video encoded data does not exist (denoted as "0" in Figure 3A). The second type is identification information that indicates that child video encoded data exists (denoted as "1" in Figure 3A). In other words, the bitstream is configured so that child video encoded data is concatenated until a "0" appears in the identification information. That is, the amount of information per identification information is 1 bit.

[0065] As shown in Figure 3A, by constructing a bitstream using hierarchical relationships, the amount of information required for identification can be reduced. For example, if the period of the audio data is 20 ms, and the time intervals (occurrence frequencies) of the video encoded data P1, P2, and P3 are 1 second, 5 seconds, and 10 seconds, respectively, the average number of bits required for identification per second would be 1 bit * 50 + 2 bits * 1 + 3 bits * 0.2 + 4 bits * 0.1 = 53 bits / second. Compared to the case explained using Figure 2, it is possible to reduce the amount of information required for identification to approximately half.

[0066] In the bitstream configuration shown in Figure 3A, the identification information is placed after the acoustic and video encoded data, but this embodiment is not limited to this. For example, as shown in Figure 3B, the identification information may be placed together at the beginning of the bitstream. The bitstreams shown in Figures 3A and 3B will be described in more detail later.

[0067] The following describes the case where audio data is treated as primary information and video data as supplementary information, as described above. In this case, audio data is always (or relatively frequently) encoded to generate encoded audio data, and video data is encoded when necessary (or relatively less frequently) to generate encoded video data.

[0068] It should be noted that this relationship between audio and video data is merely an example; video data may be considered the primary information, and audio data the supplementary information. In this case, video data is encoded at all times or at a relatively high frequency to generate encoded video data, while audio data is encoded when necessary or at a relatively low frequency to generate encoded audio data.

[0069] <First Embodiment> First, the first embodiment will be described. The first embodiment is an example of encoding and decoding a bitstream with the configuration described using Figures 3A and 3B.

[0070] [Transmitting Terminal 10] The configuration of the transmitting terminal 10 of the first embodiment will now be described. Figure 4 is a block diagram showing an example of the implementation of the transmitting terminal 10 according to the first embodiment. Figure 5 is a block diagram showing an example of the configuration of the transmitting terminal 10 according to the first embodiment.

[0071] As shown in Figure 4, the transmitting terminal 10 includes an encoding device 100. The encoding device 100 includes a processor a1 and a memory a2. For example, several components included in the transmitting terminal 10 shown in Figure 5 (described later) (including at least an acoustic data encoder 11, a video data encoder 12, and a multiplexing unit 13) are implemented by the processor a1 and memory a2 shown in Figure 4.

[0072] Processor a1 is a circuit that performs information processing and is a circuit that can access memory a2. For example, processor a1 is a dedicated or general-purpose electronic circuit for encoding sound data. Alternatively, for example, processor a1 is a dedicated or general-purpose electronic circuit for encoding video data. Processor a1 may be a processor such as a CPU. Alternatively, processor a1 may be a collection of multiple electronic circuits. Alternatively, for example, processor a1 may play the role of multiple components of the encoding device 100 shown in Figure 5, etc., described later, excluding the component for storing information.

[0073] Memory a2 is a dedicated or general-purpose memory in which information for the processor a1 to encode acoustic data, etc., is stored. Memory a2 may be an electronic circuit and may be connected to the processor a1. Memory a2 may also be included in the processor a1. Memory a2 may also be a collection of multiple electronic circuits. Memory a2 may also be a magnetic disk or an optical disk, etc., or may be described as storage or a recording medium, etc. Memory a2 may also be a non-volatile memory or a volatile memory. For example, memory a2 may store acoustic data to be encoded, or it may store a stream corresponding to encoded acoustic data. Memory a2 may also store a program for the processor a1 to encode acoustic data.

[0074] As shown in Figure 5, the transmitting terminal 10 includes an acoustic data encoder 11, a video data encoder 12, and a multiplexing unit 13.

[0075] Each of the multiple microphones 30 captures acoustic data and outputs it to the acoustic data encoder 11. Each of the multiple cameras 40 records video data according to the time resolution corresponding to the priority and outputs it to the video data encoder 12.

[0076] The acoustic data encoder 11 performs encoding processing on the input acoustic data to generate acoustic encoded data. The acoustic data encoder 11 outputs the generated acoustic encoded data to the multiplexing unit 13. Either a lossless compression method or an irreversible compression method may be used for the acoustic data encoder 11.

[0077] Similarly, the video data encoder 12 performs encoding processing on the input video data to generate encoded video data. The video data encoder 12 outputs the generated encoded video data to the multiplexing unit 13. Either a lossless compression method or an irreversible compression method may be used for the video data encoder 12.

[0078] The multiplexing unit 13 acquires acoustic coding data and video coding data. The multiplexing unit 13 multiplexes the acoustic coding data, video coding data, and identification information indicating what kind of video coding data is included to generate a bitstream. The multiplexing unit 13 outputs the bitstream to the monitoring terminal 20.

[0079] The operation of the transmitting terminal 10 will now be described. Figure 6 is a flowchart showing an example of the operation of the transmitting terminal 10 according to the first embodiment.

[0080] First, the acoustic data encoder 11 of the transmitting terminal 10 performs encoding processing on the acoustic data to generate acoustically encoded data (S101).

[0081] Next, or simultaneously, the video data encoder 12 of the transmitting terminal 10 performs encoding processing on the video data to generate video encoded data (S102).

[0082] Finally, the multiplexing unit 13 of the transmitting terminal 10 multiplexes the acoustic coding data, video coding data, and identification information into a bitstream. Here, the identification information indicates whether or not there is video coding data that occurs less frequently than the acoustic coding data or the video coding data for each of the multiple coding data, including the acoustic coding data. In other words, the identification information indicates whether or not there is video coding data that occurs less frequently than the acoustic coding data. Furthermore, the identification information indicates whether or not there is video coding data that occurs less frequently than the video coding data (i.e., child video coding data).

[0083] This encoding process by the transmitting terminal 10 generates a bitstream with the configuration shown in Figures 3A and 3B.

[0084] In the bitstream shown in Figure 3A, the identification information is located at the position indicated by the letter "A" (1 bit). Bitstream d1 shown in Figure 3A is the bitstream generated when the video encoded data is not multiplexed. Bitstream d2 is the bitstream generated when the video encoded data P1 is multiplexed. Bitstream d3 is the bitstream generated when the video encoded data P1 and P2 are multiplexed. Bitstream d4 is the bitstream generated when the video encoded data P1, P2, and P3 are multiplexed.

[0085] Depending on the generation status of each video encoding data (video encoding data P1, P2, and P3), one of the bitstreams d1 to d4 is generated.

[0086] Figure 3B corresponds to Figure 3A. In the bitstream shown in Figure 3B, the identification information is also located at the position indicated by the "A" format (1 bit). Bitstream d1' shown in Figure 3B is the bitstream generated when the video encoded data is not multiplexed. Bitstream d2' is the bitstream generated when the video encoded data P1 is multiplexed. Bitstream d3' is the bitstream generated when the video encoded data P1 and P2 are multiplexed. Bitstream d4' is the bitstream generated when the video encoded data P1, P2, and P3 are multiplexed.

[0087] Depending on the generation status of each video encoding data (video encoding data P1, P2, and P3), one of the bitstreams d1' to d4' is generated.

[0088] Furthermore, in each bitstream shown in Figures 3A and 3B, the boundaries between the audio encoded data and the video encoded data P1 to P3 (in other words, the positions where each encoded data is located) are identifiable. Specifically, the boundaries of the encoded data are identifiable because the size information of the encoded data is included in each encoded data. Alternatively, the boundaries of the encoded data are identifiable because the size information of the encoded data is held or notified to the decoding device 200 (monitoring terminal 20) in advance. Alternatively, the boundaries of the encoded data are identifiable because an termination code indicating the end of the encoded data exists at the end of each encoded data.

[0089] In this way, the transmitting terminal 10 can achieve a more efficient bitstream configuration with less bit allocation to the identification information by multiplexing the video encoded data using identification information with a reduced number of bits.

[0090] [Monitoring Terminal 20] The configuration of the monitoring terminal 20 of the first embodiment will now be described. Figure 7 is a block diagram showing an example of the implementation of the monitoring terminal according to the first embodiment. Figure 8 is a block diagram showing the functional configuration of the monitoring terminal according to the first embodiment.

[0091] As shown in Figure 7, the monitoring terminal 20 includes a decoding device 200. The decoding device 200 includes a processor b1 and memory b2. For example, several components of the monitoring terminal 20 shown in Figure 8 (described later), including at least a separation unit 21, an acoustic data decoder 22, and a video data decoder 23, are implemented by the processor b1 and memory b2 shown in Figure 7.

[0092] Processor b1 is a circuit that performs information processing and is a circuit that can access memory b2. For example, processor b1 is a dedicated or general-purpose electronic circuit that decodes streams. Processor b1 may be a processor such as a CPU. Alternatively, processor b1 may be a collection of multiple electronic circuits. Furthermore, for example, processor b1 may play the role of multiple components of the monitoring terminal 20 shown in Figure 8, etc., described later, excluding the component for storing information.

[0093] Memory b2 is a dedicated or general-purpose memory in which information for the processor b1 to decode the stream is stored. Memory b2 may be an electronic circuit and may be connected to the processor b1. Memory b2 may also be included in the processor b1. Memory b2 may also be a collection of multiple electronic circuits. Memory b2 may also be a magnetic disk or an optical disk, or may be described as storage or a recording medium. Memory b2 may also be a non-volatile memory or a volatile memory. For example, memory b2 may store acoustic data or a stream. Memory b2 may also store a program for the processor b1 to decode the stream. For example, memory b2 may also play the role of an information-storing component among the multiple components of the monitoring terminal 20 shown in Figure 8 below.

[0094] As shown in Figure 8, the monitoring terminal 20 receives the bitstreams generated by each of the multiple transmitting terminals 10. The monitoring terminal 20 includes a separation unit 21, an acoustic data decoder 22, a video data decoder 23, and a determination unit 24.

[0095] The separation unit 21 separates the audio encoded data and the video encoded data from the bitstream. The separation unit 21 outputs the separated audio encoded data to the audio data decoder 22. The separation unit 21 outputs the separated video encoded data to the video data decoder 23.

[0096] The acoustic data decoder 22 performs a decoding process on the input acoustic coding data. As a result, the acoustic data decoder 22 generates acoustic decoding data for a number of channels corresponding to the number of microphones 30. The acoustic data decoder 22 outputs the acoustic decoding data to the determination unit 24.

[0097] The video data decoder 23 performs a decoding process on the input video encoded data. As a result, the video data decoder 23 generates video decoded data for a number of channels corresponding to the number of cameras 40. The video data decoder 23 outputs the video decoded data to the determination unit 24.

[0098] The monitoring terminal 20 comprises a plurality of separation units 21, a plurality of acoustic data decoders 22, and a plurality of video data decoders 23. Specifically, the monitoring terminal 20 comprises a number of separation units 21, acoustic data decoders 22, and video data decoders 23 corresponding to the number of transmitting terminals 10.

[0099] The determination unit 24 outputs the status of the industrial robot to which the corresponding transmission terminal 10 is located as the determination result.

[0100] An example of the operation of the monitoring terminal 20 will be described. Figure 9A is a flowchart showing an example of the operation of the monitoring terminal 20 according to the first embodiment. The monitoring terminal 20 separates the bitstream that has been multiplexed by the operation described using Figure 6 by the operation shown in Figure 9A. Note that Figure 9A shows the operation applied to the bitstream with the configuration shown in Figure 3A.

[0101] As shown in Figure 9A, first, the separation unit 21 separates the acoustically encoded data (S201).

[0102] Next, the separation unit 21 determines whether the identification information indicates the existence of child video encoded data (S202). For example, the separation unit 21 determines that no child video encoded data exists if the identification information is "0", and determines that child video encoded data exists if the identification information is "1".

[0103] If the identification information determines that child video encoded data exists (Yes in S202), the separation unit 21 separates the video encoded data (S203). The separation unit 21 separates the video encoded data whose existence is indicated by the identification information.

[0104] If the identification information determines that no child video encoded data exists (No in S202), the separation unit 21 terminates its operation.

[0105] In this way, the monitoring terminal 20 can appropriately separate the video encoded data that is multiplexed in the bitstream by repeatedly separating the video encoded data until it is determined that the identification information indicates that no child video encoded data exists.

[0106] Another example of the operation of the monitoring terminal 20 will be described. Figure 9B is a flowchart of another example of the operation of the monitoring terminal 20 according to the first embodiment. Figure 9B shows the operation applied to the bitstream with the configuration shown in Figure 3B.

[0107] First, the separation unit 21 separates the identification information and interprets the separated identification information (S211). By interpreting the separated identification information, the separation unit 21 determines whether or not there is child video information included in the bitstream.

[0108] For example, when the separation unit 21 separates the sequence of "1" and "0" identification information that are arranged together from the bitstream shown in Figure 3B, it interprets that the first identification information "1" indicates that the audio encoded data has a child video encoded data P1, and the second identification information "0" indicates that the video encoded data P1 does not have a child video encoded data.

[0109] As a result, the separation unit 21 identifies that the bitstream has the same configuration as bitstream d2', and in subsequent processing, it can appropriately separate the video encoded data multiplexed in the bitstream.

[0110] Furthermore, the subsequent processes (steps S212 to S214) are the same as those described using Figure 9A, and are performed based on the identification information acquired in advance (in step S211) (that is, not on identification information acquired individually in order each time). Therefore, a further explanation is omitted here.

[0111] <Second Embodiment> Here, there may be multiple hierarchical relationships for the video encoded data included in the bitstream. The following describes this second embodiment.

[0112] In such cases, the transmitting terminal 10 and the monitoring terminal 20 maintain a common table (pattern table) that can identify patterns. The bitstream output from the transmitting terminal 10 includes pattern information, and according to this pattern information, combinations of hierarchical video encoding data types that can be included in the bitstream are identified. This makes it possible to generate bitstreams that support various combinations of video encoding data, resulting in a highly flexible bitstream. This pattern information may be shared between the transmitting terminal 10 and the monitoring terminal 20 during signaling. In that case, it becomes unnecessary to include the pattern information in the bitstream. Alternatively, the pattern information may be included in the bitstream only when the pattern information changes.

[0113] In other words, the bitstream has pattern information multiplexed onto it in addition to identification information. Pattern information is information that indicates a pattern. Multiple patterns are available, and each of these patterns indicates a combination of multiple video encoded data that is multiplexed onto the bitstream.

[0114] Figure 10 shows an example of a pattern table according to the second embodiment. In the pattern table, pattern information is associated with the pattern indicated by the pattern information. The video encoded data placed at position 1 and the video encoded data placed at position 2 have a parent-child relationship. The video encoded data placed at position 2 and the video encoded data placed at position 3 also have a parent-child relationship. The video encoded data placed at position 3 and the video encoded data placed at position 4 also have a parent-child relationship.

[0115] For example, if the pattern "01" is selected, the video encoded data that can be included in the bitstream are video encoded data P2, P3, P5, and P6, and their temporal resolutions are in the relationship P2 > P3 > P5 > P6.

[0116] In such cases, the bitstream generated by the transmitting terminal 10 will have the configuration shown below. Figure 11A is a diagram showing an example of the configuration of a bitstream including pattern information according to the second embodiment. Figure 11B is a diagram showing another example of the configuration of a bitstream including pattern information according to the second embodiment.

[0117] In the bitstreams shown in Figures 11A and 11B, the pattern information is located at the position indicated by the format "BB" (2 bits represented by a dot). The identification information is located at the position indicated by the format "A" (1 bit).

[0118] The pattern information "01" shown in Figure 11A indicates that the video encoded data that may be included in the bitstream are video encoded data P2, P3, P5, and P6.

[0119] Bitstream d11 is a bitstream generated when the video encoded data is not multiplexed. Bitstream d12 is a bitstream generated when the video encoded data P2 is multiplexed. Bitstream d13 is a bitstream generated when the video encoded data P2 and P3 are multiplexed. Bitstream d14 is a bitstream generated when the video encoded data P2, P3, and P5 are multiplexed. Bitstream d15 is a bitstream generated when the video encoded data P2, P3, and P6 are multiplexed.

[0120] Furthermore, this pattern information is not necessarily limited to fixed selections. For example, the transmitting terminal 10 can be configured to dynamically change the pattern information it references in response to the state of the monitored object or a control signal received from the monitoring terminal 20. As an example, consider a monitoring scenario for industrial equipment. In normal monitoring mode, a pattern (e.g., pattern "00") is selected to broadly grasp the state of the entire equipment. However, once a specific abnormality is detected, the transmitting terminal 10 automatically, or based on external instructions, switches to a pattern (e.g., pattern "01") defined to monitor the abnormal location in more detail. This allows for the dynamic application of the optimal data structure according to the monitoring task, maximizing both transmission efficiency and monitoring accuracy. In this way, by switching between multiple predefined hierarchical relationship patterns according to the situation, it becomes possible to adaptively redefine the hierarchical structure itself during system operation, enabling a more advanced and autonomous monitoring system.

[0121] In the bitstream configuration shown in Figure 11A, the pattern information and identification information are placed after the acoustic coding data and video coding data, but this embodiment is not limited to this. For example, as shown in Figure 11B, the pattern information and identification information may be placed together at the beginning of the bitstream. The bitstreams d11' to d15' shown in Figure 11B correspond to the bitstreams d11 to d15 shown in Figure 11A.

[0122] Next, a method for separating video encoded data contained in a bitstream that includes pattern information will be described. Figure 12A is a flowchart showing an example of the operation of the monitoring terminal 20 according to the second embodiment. Note that Figure 12A shows the operation applied to a bitstream with the configuration shown in Figure 11A.

[0123] As shown in Figure 12A, first the separation unit 21 separates the acoustically encoded data (S301).

[0124] Next, the separation unit 21 identifies a pattern based on the pattern information (S302). Specifically, the separation unit 21 identifies which pattern to use to separate the video encoded data by referring to a pattern table based on the pattern information contained in the bitstream.

[0125] Next, the separation unit 21 determines whether the identification information indicates the existence of child video encoded data (S303). For example, the separation unit 21 determines that no child video encoded data exists if the identification information is "0", and determines that child video encoded data exists if the identification information is "1".

[0126] If the identification information determines that child video encoded data exists (Yes in S303), the separation unit 21 separates the video encoded data (S304). The separation unit 21 separates the video encoded data whose existence is indicated by the identification information.

[0127] If the identification information determines that no child video encoded data exists (No in S303), the separation unit 21 terminates its operation.

[0128] In this way, the monitoring terminal 20 can appropriately separate the video encoded data multiplexed in the bitstream by repeatedly separating the video encoded data until it is determined that the identification information indicates that no child video encoded data exists. The monitoring terminal 20 can appropriately separate the video encoded data of combinations indicated by the patterns included in the pattern information.

[0129] Next, other methods for separating a bitstream containing pattern information will be described. Figure 12B is a flowchart showing another example of the operation of a monitoring terminal according to the second embodiment. Figure 12B shows the operation applied to a bitstream with the configuration shown in Figure 11B.

[0130] First, the separation unit 21 separates the pattern information and interprets the separated pattern information (S311). Specifically, the separation unit 21 identifies a pattern based on the separated pattern information. Specifically, the separation unit 21 identifies which pattern to use to separate the video encoded data by referring to a pattern table based on the pattern information contained in the bitstream.

[0131] Next, the separation unit 21 separates the identification information and interprets the separated identification information (S312). Based on the pattern information, the separation unit 21 interprets the obtained identification information and determines whether or not there is child video information included in the bitstream.

[0132] The subsequent processes (processes S313 to S316) are the same as those described using Figure 11A, and are performed based on the pattern information and identification information acquired in advance (in the processes of steps S311 and S312). Therefore, a further explanation is omitted here.

[0133] <Third Embodiment> In the above embodiment, the identification information indicated whether or not child video encoded data existed. Here, the identification information may also be information indicating that specific video encoded data is multiplexed. The following describes the information processing system 1 of this third embodiment.

[0134] The configuration of the transmitting terminal 10 and the monitoring terminal 20 according to the third embodiment is the same as the configuration of the transmitting terminal 10 and the monitoring terminal 20 described with reference to Figures 5 and 8 in the first embodiment.

[0135] In the transmitting terminal 10, the identification information includes an exception code, and when an exception code appears, specific video encoded data is multiplexed into a bitstream.

[0136] In such cases, the bitstream generated by the transmitting terminal 10 will have the configuration shown below. Figure 13A is a diagram showing an example of the configuration of a bitstream including an exception code according to the third embodiment. Figure 13B is a diagram showing another example of the configuration of a bitstream including an exception code according to the third embodiment.

[0137] The video encoding data PX1 in Figures 13A and 13B is specific video encoding data.

[0138] In the bitstreams shown in Figures 13A and 13B, the exception code is located at the position indicated by the letter "C" (1 bit indicated by a diagonal line). The identification information is located at the position indicated by the letter "AA" (2 bits).

[0139] As shown in Figure 13A, the identification information is "1" when it indicates that "specific video encoded data" exists. Hereafter, such identification information will be referred to as an exception code. The identification information is "00" when it indicates that "child video encoded data" does not exist. The identification information is "01" when it indicates that "child video encoded data" exists. In other words, the identification information changes because it is necessary to distinguish between three situations.

[0140] Bitstream d21 is a bitstream generated when the video encoded data is not multiplexed. Bitstream d22 is a bitstream generated when the video encoded data P1 is multiplexed. Bitstream d23 is a bitstream generated when the video encoded data P1 and P2 are multiplexed. Bitstream d24 is a bitstream generated when the video encoded data P1, P2, and P3 are multiplexed. Bitstream d25 is a bitstream generated when the video encoded data PX1 is multiplexed. Bitstream d26 is a bitstream generated when the video encoded data P1 and PX1 are multiplexed.

[0141] In bitstream d25, specific video encoded data PX1 is multiplexed following the audio encoded data. In this case, an exception code "1" is assigned as identification information so that the monitoring terminal 20 can recognize that specific video encoded data PX1 is being multiplexed. Figure 13A shows the bitstream configuration when there is no video encoded data that is a child of the video encoded data PX1. In this case, when there is no child video encoded data for the video encoded data PX1, it is not necessary to place identification information "00" indicating that there is no child video encoded data following the video encoded data PX1, or identification information "01" indicating that there is child video encoded data. In other words, in the example shown in Figure 13A, the exception code "1" also serves as identification information indicating that there is no video encoded data other than the video encoded data PX1.

[0142] In bitstream d26, video encoded data P1 and PX1 are multiplexed. By using an exception code as identification information for multiplexing specific video encoded data PX1 in this way, it is also possible to multiplex it together with normal video encoded data.

[0143] In the bitstream configuration shown in Figure 13A, the identification information including the exception code is placed after the acoustic coding data and the video coding data, but this embodiment is not limited to this. For example, as shown in Figure 13B, the identification information including the exception code may be placed together at the beginning of the bitstream. The bitstreams d21' to d26' shown in Figure 13B are the bitstreams corresponding to the bitstreams d21 to d26 shown in Figure 13A.

[0144] Next, a method for separating video encoded data contained in a bitstream that includes an exception code will be described. Figure 14A is a flowchart showing an example of the operation of the monitoring terminal 20 according to the third embodiment. Note that Figure 14A shows the operation applied to a bitstream with the configuration shown in Figure 13A.

[0145] As shown in Figure 14A, first the separation unit 21 separates the acoustically encoded data (S401).

[0146] Next, the separation unit 21 determines whether the identification information is an exception code (S402). The separation unit 21 determines that the identification information is an exception code if the identification information is "1", and determines that the identification information is not an exception code if the identification information is not "1" (i.e., if it is "01" or "00").

[0147] If the identification information is determined not to be an exception code (No in S402), the separation unit 21 determines whether the identification information indicates the existence of child video encoded data (S403). For example, if the identification information is "00", the separation unit 21 determines that no child video encoded data exists, and if the identification information is "01", it determines that child video encoded data exists.

[0148] If the identification information determines that child video encoded data exists (Yes in S403), the separation unit 21 separates the video encoded data (S404). The separation unit 21 separates the video encoded data whose existence is indicated by the identification information.

[0149] If the identification information determines that no child video encoded data exists (No in S403), the separation unit 21 terminates its operation. In this way, the monitoring terminal 20 repeats the separation of video encoded data until it determines that the identification information indicates that no child video encoded data exists.

[0150] Furthermore, if the identification information is determined to be an exception code (Yes in S402), the separation unit 21 separates a specific parameter (for example, video encoding data PX1) (S405). After that, the separation unit 21 terminates its operation.

[0151] In this way, the monitoring terminal 20 can appropriately separate the video encoded data multiplexed into a bitstream that includes an exception code.

[0152] Next, other methods for isolating bitstreams containing exception codes will be described. Figure 14B is a flowchart showing another example of the operation of a monitoring terminal according to the third embodiment. Figure 14B shows the operation applied to a bitstream with the configuration shown in Figure 13B.

[0153] First, the separation unit 21 separates the identification information and interprets the separated identification information (S411). The separation unit 21 interprets the identification information and determines whether or not there is child video information included in the bitstream.

[0154] The subsequent processes (steps S412 to S416) are the same as those described using Figure 14A, and are performed based on the identification information acquired in advance (in step S411). Therefore, a further explanation is omitted here.

[0155] On the other hand, there may be video encoded data that is a child of the video encoded data PX1. In such cases, the bitstream generated by the transmitting terminal 10 will have the configuration shown below. Figure 15A is a diagram showing another example of the configuration of a bitstream including an exception code according to the third embodiment. Figure 15B is a diagram showing yet another example of the configuration of a bitstream including an exception code according to the third embodiment.

[0156] In Figures 15A and 15B, the video encoded data PX2 is a child video encoded data of the video encoded data PX1. In the bitstream shown in Figures 15A and 15B, the exception code is located at the position indicated by the letter "C" (1 bit indicated by a diagonal line). The identification information is located at the position indicated by the letter "AA" (2 bits).

[0157] Bitstream d31, shown in Figure 15A, is a bitstream generated when the video encoded data is not multiplexed. Bitstream d32 is a bitstream generated when the video encoded data P1 is multiplexed. Bitstream d33 is a bitstream generated when the video encoded data P1 and P2 are multiplexed. Bitstream d34 is a bitstream generated when the video encoded data P1, P2, and P3 are multiplexed. Bitstream d35 is a bitstream generated when the video encoded data PX1 is multiplexed. Bitstream d36 is a bitstream generated when the video encoded data P1, PX1, and PX2 are multiplexed.

[0158] In bitstream d35, specific video encoded data PX1 is multiplexed following the audio encoded data. In this case, an exception code "1" is assigned as identification information so that the monitoring terminal 20 can recognize that the video encoded data PX1 is being multiplexed. In bitstream d35, the child video encoded data PX2 is not multiplexed to the specific video encoded data PX1. Therefore, "00", which is identification information indicating that there is no child video encoded data following the video encoded data PX1, is assigned.

[0159] In bitstream d36, video encoded data P1, specific video encoded data PX1, and its child video encoded data PX2 are multiplexed. At this time, the identifier "01" is assigned to video encoded data PX1 to indicate that there is child video encoded data following it. Next, since the child video encoded data of video encoded data PX2 is not subject to multiplexing, the identifier "00" is assigned to video encoded data PX2 to indicate that there is no child video encoded data following it.

[0160] In the bitstream configuration shown in Figure 15A, the identification information including the exception code is placed after the acoustic coding data and the video coding data, but this embodiment is not limited to this. For example, as shown in Figure 15B, the identification information including the exception code may be placed together at the beginning of the bitstream. The bitstreams d31' to d36' shown in Figure 15B are the bitstreams corresponding to the bitstreams d31 to d36 shown in Figure 15A.

[0161] Next, we will describe another method for separating video encoded data contained in a bitstream that includes an exception code. Figure 16A is a flowchart showing another example of the operation of the monitoring terminal 20 according to the third embodiment. Figure 16A shows the operation applied to a bitstream with the configuration shown in Figure 15A.

[0162] As shown in Figure 16A, first the separation unit 21 separates the acoustically encoded data (S501).

[0163] Next, the separation unit 21 determines whether the identification information is an exception code (S502). The separation unit 21 determines that the identification information is an exception code if the identification information is "1", and determines that the identification information is not an exception code if the identification information is not "1" (i.e., if it is "01" or "00").

[0164] If the identification information is determined not to be an exception code (No in S502), the separation unit 21 determines whether the identification information indicates the existence of child video encoded data (S503). For example, if the identification information is "00", the separation unit 21 determines that no child video encoded data exists, and if the identification information is "01", it determines that child video encoded data exists.

[0165] Furthermore, if the identification information is determined to be an exception code (Yes in S502), the separation unit 21 separates a specific parameter (for example, video encoded data PX1 or video encoded data PX2) (S505). Then, proceeding to step S503, the separation unit 21 determines whether or not the identification information indicates the existence of child video encoded data.

[0166] If the identification information determines that child video encoded data exists (Yes in S503), the separation unit 21 separates the video encoded data (S504). The separation unit 21 separates the video encoded data whose existence is indicated by the identification information.

[0167] If the identification information determines that no child video encoded data exists (No in S503), the separation unit 21 terminates its operation. In this way, the monitoring terminal 20 repeats the separation of video encoded data until it determines that the identification information indicates that no child video encoded data exists.

[0168] In this way, the monitoring terminal 20 can appropriately separate the video encoded data multiplexed into a bitstream that includes the exception code.

[0169] Next, we will describe another method for isolating a bitstream containing an exception code. Figure 16B is a flowchart showing yet another example of the operation of a monitoring terminal according to the third embodiment. Figure 16B shows the operation applied to a bitstream with the configuration shown in Figure 15B.

[0170] First, the separation unit 21 separates the identification information and interprets the separated identification information (S511). The separation unit 21 interprets the identification information and determines whether or not there is child video information included in the bitstream.

[0171] The subsequent processes (steps S512 to S516) are the same as those described using Figure 16A, and are performed based on the identification information acquired in advance (in step S511). Therefore, a further explanation is omitted here.

[0172] <Fourth Embodiment> In the above embodiment, the video encoded data multiplexed into the bitstream is in a hierarchical relationship, and if the identification information indicates the existence of child video encoded data, the child video encoded data is always multiplexed into the bitstream. However, the child video encoded data does not necessarily have to be multiplexed.

[0173] The following describes this fourth embodiment. In the fourth embodiment, supplementary information is introduced. Supplementary information indicates whether or not the child video encoded data is multiplexed into the bitstream. That is, when the supplementary information is 0, the child video encoded data is multiplexed into the bitstream as in the previous embodiments, and when the supplementary information is 1, the child video encoded data is not multiplexed into the bitstream.

[0174] In such cases, the bitstream generated by the transmitting terminal 10 will have the configuration shown below. Figure 17A is a diagram showing an example of the configuration of a bitstream including supplementary information according to the fourth embodiment. Figure 17B is a diagram showing another example of the configuration of a bitstream including supplementary information according to the fourth embodiment.

[0175] In the bitstreams shown in Figures 17A and 17B, the interpolation information is located at the position indicated by the "D" format (1 bit indicated by a stripe). The identification information is located at the position indicated by the "A" format (1 bit).

[0176] Bitstream d41, shown in Figure 17A, is a bitstream generated when the video encoded data is not multiplexed. Bitstream d42 is a bitstream generated when the video encoded data P1 is multiplexed. Bitstream d43 is a bitstream generated when the video encoded data P1 and P2 are multiplexed. Bitstream d44 is a bitstream generated when the video encoded data P1, P2, and P3 are multiplexed. Bitstream d45 is a bitstream generated when the video encoded data P1 and P3 are multiplexed.

[0177] As shown in bitstreams d43 and d44, when the interpolation information is "0", the child video encoded data (video encoded data P2) is multiplexed into the bitstream, as in previous embodiments.

[0178] As shown in bitstream d45, when the interpolation information is "1", the video encoded data P2 is interpolated and not multiplexed into the bitstream. In other words, the multiplexing of the video encoded data P2 is omitted.

[0179] In the bitstream configuration shown in Figure 17A, the identification information and complementary information are placed after the acoustic coding data and the video coding data, but this embodiment is not limited to this. For example, as shown in Figure 17B, the identification information and complementary information may be placed together at the beginning of the bitstream. The bitstreams d41' to d45' shown in Figure 17B are bitstreams corresponding to the bitstreams d41 to d45 shown in Figure 17A.

[0180] Next, a method for separating video encoded data contained in a bitstream that includes complementary information will be described. Figure 18A is a flowchart showing an example of the operation of the monitoring terminal 20 according to the fourth embodiment. Figure 18A shows the operation applied to a bitstream with the configuration shown in Figure 17A.

[0181] As shown in Figure 18A, first the separation unit 21 separates the acoustically encoded data (S601).

[0182] Next, the separation unit 21 determines whether the identification information indicates the existence of child video encoded data (S602). For example, the separation unit 21 determines that no child video encoded data exists if the identification information is "0", and determines that child video encoded data exists if the identification information is "1".

[0183] If the identification information is determined to indicate the existence of child video encoded data (Yes in S602), the separation unit 21 determines whether the supplementation information indicates that the corresponding encoded data has been supplemented (S603). The separation unit 21 refers to the supplementation information and determines that if the supplementation information is "0", it indicates that the corresponding encoded data has not been supplemented, and if the supplementation information is "1", it indicates that the corresponding encoded data has been supplemented.

[0184] If the supplementation information determines that the corresponding encoded data is not supplemented (No in S603), the separation unit 21 separates the video encoded data that the identification information indicates is present (S604).

[0185] If it is determined that the supplementation information indicates that the corresponding encoded data has been supplemented (Yes in S603), the process returns to step S602, and the separation unit 21 determines whether or not the identification information indicates that child video encoded data exists.

[0186] If the identification information determines that no child video encoded data exists (No in S602), the separation unit 21 terminates its operation. In this way, the monitoring terminal 20 repeats the separation of video encoded data until it determines that the identification information indicates that no child video encoded data exists.

[0187] In this way, the monitoring terminal 20 can appropriately separate the video encoded data that has been multiplexed into a bitstream containing supplementary information.

[0188] Next, other methods for separating a bitstream containing complementary information will be described. Figure 18B is a flowchart showing another example of the operation of the monitoring terminal 20 according to the fourth embodiment. Figure 18B shows the operation applied to a bitstream with the configuration shown in Figure 17B.

[0189] First, the separation unit 21 separates the identification information and the supplementary information, and interprets the separated identification information and supplementary information (S611). Based on the separated identification information and supplementary information, the separation unit 21 determines whether or not there is child video information included in the bitstream.

[0190] The subsequent processes (steps S612 to S615) are the same as those described using Figure 18A, and are performed based on the identification information and supplementary information acquired in advance (in step S611). Therefore, a further explanation is omitted here.

[0191] By introducing complementary information in this way, even if the video encoded data P2 is not included in the bitstream, it becomes possible to include the video encoded data P3, which is a child of the video encoded data P2, in the bitstream. In other words, it becomes possible to construct a bitstream that includes video encoded data P1 and P3 other than the video encoded data P2, as shown in bitstream d45 in Figure 17A.

[0192] Thus, according to the fourth embodiment, it becomes possible to configure a bitstream with even greater flexibility while reducing the amount of information in the identification information. The fourth embodiment can reduce the amount of information in the bitstream.

[0193] Furthermore, if the monitoring terminal 20 detects the supplementary information "1" in the bitstream, it may recognize that the corresponding child video encoded data is not included in the bitstream, perform a decoding process using the corresponding child video encoded data received in the past, generate decoded video data, and use it for determination.

[0194] The following describes another method for separating video encoded data contained in a bitstream that includes such complementary information. Figure 19A is a flowchart showing another example of the operation of a monitoring terminal according to the fourth embodiment. Figure 19A shows the operation applied to a bitstream with the configuration shown in Figure 17A.

[0195] The processes in steps S701 to S704 in Figure 19A are the same as those in steps S601 to S604 in Figure 18A, so a further explanation is omitted here.

[0196] As shown in Figure 19A, if it is determined that the supplementation information indicates that the corresponding encoded data has been supplemented (Yes in S703), the separation unit 21 uses the corresponding child video encoded data that was received in the past (S705). For example, if it is determined that the encoded data corresponding to the supplementation information has been supplemented, the separation unit 21 uses the corresponding child video encoded data that was included in another bitstream received in the past as the supplemented child video encoded data. Then, returning to the process in step S702, the separation unit 21 determines whether or not the identification information indicates that child video encoded data exists.

[0197] Figure 19B is a flowchart showing yet another example of the operation of the monitoring terminal according to the fourth embodiment. Figure 19B shows the operation applied to the bitstream with the configuration shown in Figure 17B.

[0198] The processes in steps S711 to S715 in Figure 19B are the same as those in steps S611 to S615 in Figure 18B, so a further explanation is omitted here.

[0199] As shown in Figure 19B, if it is determined that the supplementation information indicates that the corresponding encoded data has been supplemented (Yes in S714), the separation unit 21 uses the corresponding child video encoded data that was received in the past (S716). For example, if it is determined that the encoded data corresponding to the supplementation information has been supplemented, the separation unit 21 uses the corresponding child video encoded data that was included in another bitstream received in the past as the supplemented child video encoded data. Then, returning to the process in step S713, the separation unit 21 determines whether or not the identification information indicates that child video encoded data exists.

[0200] This makes it possible to decode the bitstream after supplementing it with the newly encoded data.

[0201] (Other Embodiments) The second, third, and fourth embodiments described above can be combined in any way. This supports various combinations of video encoding data and enables the generation of bitstreams containing specific video encoding data, thereby enabling the configuration of even more flexible bitstreams.

[0202] In the above embodiment, for the sake of ease of understanding, examples were used in which the identification information (whether or not it includes an exception code), pattern information, and supplementary information are represented by 1 to 2 bits. However, actual implementations are not limited to this. For example, each of the identification information, pattern information, and supplementary information may be assigned a code that allows for the determination of which information it is.

[0203] Furthermore, although the above embodiment describes audio data as primary information and video data as supplementary information, this disclosure may, regardless, describe video data as primary information and audio data as supplementary information. Such differences in treatment will depend on the application to which it is applied and the nature of the problem to be solved.

[0204] Furthermore, the data covered by this disclosure is not limited to acoustic data or video data; a wide variety of data can be treated as primary data and supplementary data in this invention. Specifically, information such as metadata indicating the state of a sensor (e.g., sensor type, location information), metadata indicating the measurement environment (e.g., temperature, humidity, atmospheric pressure), metadata indicating the state of a process (e.g., machine operation status OK / NOK, process information), metadata indicating the results of an analysis (e.g., speech recognition text, speaker ID, timestamp of anomaly detection), biosignal data (e.g., waveform data such as electrocardiogram (ECG) and electroencephalogram (EEG)), and physical sensor signal data (e.g., time-series signals obtained from speed sensors, acceleration sensors, illuminance sensors, humidity sensors, and temperature sensors) can be hierarchically multiplexed as primary data and supplementary data.

[0205] Thus, this disclosure is a technology that efficiently multiplexes not only raw signal data, but also various metadata indicating their acquisition conditions and analysis results, or multidimensional streams such as medical data and various sensor signals, based on hierarchical relationships, enabling a wide range of industrial applications.

[0206] Furthermore, the hierarchical relationships can be set according to the type of media, such as whether it is audio data or video data, or regardless of the type of media, based on the physical characteristics of the signal, such as whether the data changes significantly over time or only slightly. The hierarchical relationships can also be set based on the location of sensors such as microphones and cameras, or acquisition conditions such as the direction of directivity for microphones or the framing location for cameras. Additionally, the hierarchical relationships can be set according to the magnitude of the influence these data have on the decision-making process at the monitoring terminal.

[0207] For example, in a predictive maintenance scenario for industrial equipment, one use case, normal operating noise data with low impact on the decision-making process is set as primary data, while specific vibration noise data or temperature data indicating anomalies with a very high impact on the decision-making process are set as supplementary data. In this case, the primary data becomes the "parent" data that is transmitted at all times, and the supplementary data becomes the "child" data that is transmitted only when an anomaly is detected. In this way, by designating highly important data that triggers the decision as "child" data and the contextual data that serves as its premise as the "parent" data, a hierarchical relationship based on the impact on the decision-making process can be established. This makes it possible to suppress the amount of data transmitted during normal operation while efficiently transmitting the information necessary for the decision-making process when it occurs.

[0208] Furthermore, the data subject to multiplexing in this invention is not limited to raw signals directly encoded from sensors or the like. "Features" extracted through intermediate processing such as machine learning models can also be treated as encoded data in this invention. For example, a configuration can be considered in which lighter, more abstract features are used as "parent" data, and high-dimensional, detailed features are hierarchically multiplexed as "child" data only when a specific event is detected or when more detailed analysis is required. Specifically, a spectrogram overview of the acoustic signal is used as parent data that is transmitted at all times, and detailed feature vectors corresponding to the time-frequency domain in which abnormal sounds are detected are added as child data. This makes it possible to efficiently transmit the core information necessary for judgment while significantly reducing the amount of data compared to transmitting the raw signal itself.

[0209] In this way, by layering not only the raw signals but also the features extracted therefrom, and applying the multiplexing technology of the present invention, it is possible to efficiently realize an advanced machine learning-based monitoring system even in environments where the computing resources and network bandwidth of edge devices are limited.

[0210] Furthermore, although the above embodiment was described assuming a configuration with one monitoring terminal 20, the monitoring terminal 20 may be composed of two or more units. In such a case, each monitoring terminal 20 can monitor different tasks.

[0211] The bitstream generated by the transmitting terminal 10 and provided to these monitoring terminals 20 can have different types of data multiplexed on it, depending on the content of the task of each monitoring terminal 20. The multiplexed data includes acoustic coded data, video coded data, and coded data obtained by coding observed values ​​acquired from accelerometers, illuminance sensors, humidity sensors, temperature sensors, biosensors, etc., depending on the content of the task. By generating this coded data in a hierarchical relationship, it becomes possible to reduce the amount of information in the identification information, which is a feature of the present invention.

[0212] Furthermore, the transmitting terminal 10 duplicates the generated encoded data as needed to construct different types of bitstreams corresponding to each monitoring terminal 20, and transmits the corresponding bitstream to each monitoring terminal 20. By duplicating the encoded data in this way, it becomes possible to efficiently construct different types of bitstreams.

[0213] Furthermore, the encoded data multiplexed into the bitstream may have scalability, and the bitstream may be constructed by discarding a portion of the encoded data according to the accuracy of the decoded data required by each monitoring terminal. This allows the bitstream to be constructed based on encoded data with the minimum accuracy required by each monitoring terminal, thus providing the effect of further reducing the amount of data in the bitstream, in addition to the effect of reducing the amount of identification information, which is a feature of the present invention. Note that "encoded data with scalability" here refers to encoded data that has the property of generating highly accurate decoded data when all of the encoded data is used for decodement, and generating decoded data with reduced accuracy even when a portion of the encoded data is discarded. The greater the amount discarded, the lower the accuracy of the decoded data.

[0214] Furthermore, this disclosure also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art could conceive, or forms realized by arbitrarily combining the components and functions of each embodiment without departing from the spirit of this disclosure.

[0215] This disclosure can be used in technologies for multiplexing multiple types of information to send and receive bitstreams.

[0216] 1 Information processing system 2 Space 10 Transmitting terminal 11 Acoustic data encoder 12 Video data encoder 13 Multiplexing unit 20 Monitoring terminal 21 Separation unit 22 Acoustic data decoder 23 Video data decoder 24 Judgment unit 30 Microphone 40 Camera 50 Wireless network 100 Encoding device 200 Decoding device a1, b1 Processor a2, b2 Memory d1, d2, d3, d4, d11, d12, d13, d14, d15, d21, d22, d23, d24, d25, d26, d31, d32, d33, d34, d35, d36, d41, d42, d43, d44, d45, d1', d2', d3', d4', d11', d12', d13', d14', d15', d21', d22', d23', d24', d25', d26', d31', d32', d33', d34', d35', d36', d41', d42', d43', d44', d45' Bitstream OBJ Target

Claims

1. An encoding method comprising: encoding an input signal to generate a first encoded data and a plurality of second encoded data having a lower occurrence frequency than the first encoded data and having different occurrence frequencies from one another; multiplexing the first encoded data, the plurality of second encoded data, and identification information into a bitstream; and indicating whether or not there is a second encoded data having a lower occurrence frequency than the encoded data for each of the plurality of encoded data including the first encoded data and the plurality of second encoded data.

2. The encoding method according to claim 1, wherein pattern information, first encoded data, a plurality of second encoded data representing combinations shown in the pattern indicated by the pattern information, and identification information are multiplexed into the bitstream, and the pattern information is information representing a plurality of patterns, each representing a combination of the plurality of second encoded data multiplexed into the bitstream.

3. The encoding method according to claim 1 or 2, wherein the identification information includes an exception code, and the exception code indicates, instead of indicating whether or not there is a second encoded data which occurs less frequently than each of the plurality of encoded data, there is a third encoded data which is different from the second encoded data.

4. The encoding method according to claim 1 or 2, wherein the supplementary information, the first encoded data, the plurality of second encoded data, and the identification information are multiplexed into the bitstream, and the supplementary information is information indicating whether or not supplementation is performed on the second encoded data to be multiplexed into the bitstream.

5. A decoding method comprising decoding a bitstream to separate a first encoded data, a plurality of second encoded data having a lower occurrence frequency than the first encoded data and having different occurrence frequencies from one another, and identification information, wherein the identification information indicates whether or not there is a second encoded data having a lower occurrence frequency than the encoded data for each of the plurality of encoded data including the first encoded data and the plurality of second encoded data.

6. The decoding method according to claim 5, wherein the bitstream is decoded to separate pattern information, the first encoded data, the plurality of second encoded data representing combinations shown in the pattern indicated by the pattern information, and the identification information, wherein the pattern information is information representing a plurality of patterns, each representing a combination of the plurality of second encoded data multiplexed in the bitstream.

7. The decoding method according to claim 5 or 6, wherein the identification information includes an exception code, and the exception code indicates, instead of indicating whether or not there is a second encoded data which occurs less frequently than each of the plurality of encoded data, the identification information indicates the existence of a third encoded data which is different from the second encoded data.

8. The decoding method according to claim 5 or 6, wherein the bitstream is decoded to separate the supplementation information, the first encoded data, the plurality of second encoded data, and the identification information, the supplementation information being information indicating whether or not the second encoded data to be multiplexed in the bitstream is supplemented.

9. A program for causing a computer to execute the encoding method described in claim 1 or 2.

10. A program for causing a computer to execute the decoding method described in claim 5 or 6.

11. Encoding device comprising a circuit and a memory connected to the circuit, wherein the circuit, in operation, encodes an input signal to generate a first encoded data and a plurality of second encoded data having a lower occurrence frequency than the first encoded data and having different occurrence frequencies from one another, multiplexes the first encoded data, the plurality of second encoded data, and identification information to generate a bitstream, and the identification information indicates whether or not there is a second encoded data having a lower occurrence frequency than the encoded data for each of the plurality of encoded data including the first encoded data and the plurality of second encoded data.

12. A decoding device comprising a circuit and a memory connected to the circuit, wherein the circuit, in operation, decodes a bitstream to separate a first encoded data, a plurality of second encoded data having a lower occurrence frequency than the first encoded data and having different occurrence frequencies from each other, and identification information, wherein the identification information indicates whether or not there is a second encoded data having a lower occurrence frequency than the encoded data for each of the plurality of encoded data including the first encoded data and the plurality of second encoded data.

Images