Transmission device and transmission method

Abstract

A relay device 20 comprises: an acquisition unit 21 that acquires first video information and a plurality of pieces of state information or a plurality of pieces of intensity information; an estimation unit 22 that estimates a plurality of pieces of terminal band information indicating an available band for each of communication paths P, on the basis of the plurality of pieces of state information or the plurality of pieces of intensity information; a classification unit 23 that classifies a plurality of terminals 30 into a plurality of groups on the basis of the plurality of pieces of terminal band information; a generation unit 24 that generates a plurality of pieces of second video information indicating a plurality of second target images corresponding to the respective groups, on the basis of the first video information and a plurality of pieces of group band information indicating available bands corresponding to the plurality of groups; and a transmission unit 25 that transmits, to each of the plurality of terminals 30, second video information corresponding to a group into which said each of the plurality of terminals 30 is classified, among the plurality of pieces of second video information.

Description

Transmission device and transmission method

[0001] One aspect of the present invention relates to a transmission device and a transmission method.

[0002] Conventionally, a terminal device that transmits video data to a server device via a network is known (for example, see Patent Document 1). For example, in the terminal device described in Patent Document 1, adjustment of image data in the video data is performed based on band information associated with information indicating a network state, so that the band used by the terminal device is adjusted to a target band, and the video data is transmitted to the server device in this adjusted state.

[0003] Japanese Unexamined Patent Application Publication No. 2014-175959

[0004] However, in the terminal device as described above, when adjustment of image data in video data is performed for each destination terminal, the processing load on the terminal device becomes enormous. On the other hand, after the image data in the video data is adjusted, if the same video data is transmitted to all destination terminals, when the data amount of the video data exceeds the available band of the network between the terminal device and the destination terminal, some destination devices may not be able to smoothly play the moving image, and there is a risk that the video data will be interrupted during playback. Accordingly, it is desired to enable smooth playback of moving images on the destination device while reducing the processing load on the source device.

[0005] One aspect of the present disclosure has been made in view of the above circumstances, and an object thereof is to provide a transmission device and a transmission method that enable smooth playback of moving images on a destination device while reducing the processing load on a source device.

[0006] A transmitting device according to one aspect of the present disclosure is a transmitting device for transmitting video to multiple terminals, comprising: an acquisition unit that acquires first video information indicating a first target image which is an image to be compressed among a plurality of images included in the video; a plurality of state information indicating the state of each communication path between the transmitting device and each of the plurality of terminals, or a plurality of intensity information relating to the radio wave intensity at the plurality of terminals; an estimation unit that estimates a plurality of terminal bandwidth information indicating the available bandwidth of each communication path based on the plurality of state information or the plurality of intensity information; a classification unit that classifies the plurality of terminals into one of a plurality of groups based on the plurality of terminal bandwidth information; a generation unit that acquires a plurality of group bandwidth information indicating each available bandwidth corresponding to each of the plurality of groups, and generates a plurality of second video information indicating a plurality of second target images corresponding to each of the plurality of groups based on the plurality of group bandwidth information and the first video information; and a transmitting unit that transmits to each of the plurality of terminals the second video information corresponding to the group to which each of the plurality of terminals is classified from among the plurality of second video information.

[0007] According to one aspect of the present disclosure, a transmission device acquires a plurality of group bandwidth information indicating the available bandwidth corresponding to each of a plurality of groups, generates a plurality of second video information indicating a plurality of second target images corresponding to each of the plurality of groups based on the plurality of group bandwidth information and first video information, and transmits to each of the plurality of terminals the second video information corresponding to the group to which each of the plurality of terminals is classified from among the plurality of second video information. With this configuration, the plurality of second video information to be transmitted to each of the plurality of terminals is generated taking into account the available bandwidth of the communication path between the transmission device and each of the plurality of terminals. This makes it possible to smoothly play the video on each of the plurality of terminals. Furthermore, with the above configuration, it is possible to reduce the processing load on the transmission device compared to the case in which second video information is generated for each terminal. As a result, it is possible to smoothly play the video on the receiving device while reducing the processing load on the transmitting device.

[0008] A transmission method relating to one aspect of the present disclosure is a transmission method performed by a transmitting device, comprising: an acquisition step in which the transmitting device acquires first video information indicating a first target image which is an image to be compressed among a plurality of images included in a video, a plurality of state information indicating the state of each communication path between the transmitting device and each of the plurality of terminals, or a plurality of intensity information relating to the radio wave intensity at the plurality of terminals; an estimation step in which the transmitting device estimates a plurality of terminal bandwidth information indicating the available bandwidth of each communication path based on the plurality of state information or the plurality of intensity information; a classification step in which the transmitting device classifies the plurality of terminals into one of a plurality of groups based on the plurality of terminal bandwidth information; a generation step in which the transmitting device acquires a plurality of group bandwidth information indicating each available bandwidth corresponding to each of the plurality of groups, and generates a plurality of second video information corresponding to each of the plurality of groups based on the plurality of group bandwidth information and the first video information; and a transmission step in which the transmitting device transmits to each of the plurality of terminals the second video information corresponding to the group to which each of the plurality of terminals has been classified, from among the plurality of second video information.

[0009] According to this disclosure, it is possible to reduce the processing load on the transmitting device while enabling smooth playback of video on the receiving device.

[0010] Figure 1 is a schematic diagram showing an overview of an information processing system 1 equipped with a relay device according to one embodiment. Figure 2 is a block diagram showing the functional configuration of a transmitting terminal, a relay device, and a plurality of terminals. Figure 3 is a diagram illustrating the process of estimating terminal bandwidth information. Figure 4 is a flowchart showing an example of the transmission method processing performed by the relay device according to one embodiment. Figure 5 is a diagram showing the hardware configuration of the relay device according to the embodiment.

[0011] Embodiments of this disclosure will be described in detail below with reference to the attached drawings. In the description of the drawings, the same reference numerals are used for identical or equivalent elements, and redundant descriptions are omitted.

[0012] Figure 1 is a schematic diagram showing an overview of an information processing system 1 equipped with a relay device 20 (transmitting device) according to one embodiment. The information processing system 1 comprises a transmitting terminal 10, a relay device 20, and a plurality of terminals 30. The information processing system 1 is a system for transmitting video to the plurality of terminals 30. The transmitting terminal 10 transmits video to the relay device 20. The relay device 20 adjusts the video received from the transmitting terminal 10 and outputs it to the plurality of terminals 30. The plurality of terminals 30 display and play the received video on a display or the like.

[0013] For example, information processing system 1 is a system used by businesses that provide videos to users utilizing multiple terminals 30. Transmitting terminal 10 is a terminal used by businesses or individuals providing videos (for example, broadcasters who distribute content to viewers using live streaming platforms or social media). Relay device 20 is a server (one example being a cloud server) owned by a company that acts as an intermediary for video distribution. Multiple terminals 30 are terminals owned by each of the multiple users, such as 5G-enabled terminals and personal computers (PCs).

[0014] The transmitting terminal 10 includes a camera 101, a control device 102, and a transmitting / receiving terminal 103. The camera 101 is a device for capturing video. The control device 102 is an encoder for compressing video. The transmitting / receiving terminal 103 is, for example, a terminal or router capable of 5G communication.

[0015] Camera 101 outputs the captured and generated video to control device 102. Control device 102 compresses the video output from camera 101 (for example, by adjusting the video data size, video resolution, and video frame rate, etc.) and outputs it to transmitting / receiving terminal 103. Transmitting / receiving terminal 103 transmits the video output from control device 102 to relay device 20 via the 5G network, for example using an application that transmits live video.

[0016] The relay device 20 includes an estimation server 201 and a distribution server 202. The estimation server 201 estimates the available bandwidth of the network. For example, the estimation server 201 estimates the available bandwidth of the network between the transmitting terminal 10 and the relay device 20, and controls the control device 102 of the transmitting terminal 10 so that the amount of video data transmitted from the transmitting terminal 10 does not exceed the available bandwidth of the network. The distribution server 202 decodes the video received from the transmitting terminal 10. The distribution server 202 performs editing and recompression of the video (adjusting the aspect ratio and frame rate), etc., in order to provide a video suitable for the user viewing the video and the terminal owned by the user. For example, the distribution server 202 performs recompression of the video using multiple encoders E. The distribution server 202 transmits the video to each of the multiple terminals 30. The multiple encoders E are encoders that compress the video using a lossy encoding scheme, and examples include H.265 / HEVC, H. This encoder supports encoding schemes such as 264 / AVC, MPEG-4, MPEG-2, AV1 / VP10, and VP9.

[0017] Multiple terminals 30 play back the video received from the relay device 20 using a video playback and viewing player. The multiple terminals 30 are connected to the relay device 20 via wireless or wired connections.

[0018] The information processing system 1 enables businesses providing video to users utilizing multiple terminals 30 to receive and play video in real time. Furthermore, by using a 5G network for communication, high-speed and low-latency video transmission and control can be achieved. The functional configurations of the transmitting terminal 10, the relay device 20, and the multiple terminals 30 will be described in detail below.

[0019] Figure 2 is a block diagram showing the functional configuration of the transmitting terminal 10, the relay device 20, and the multiple terminals 30. The transmitting terminal 10 is composed of an imaging unit 11, an acquisition unit 12, an adjustment unit 13, and a transmitting unit 14 as its functional components. The imaging unit 11 corresponds to the camera 101 described above, the acquisition unit 12 and the transmitting unit 14 correspond to the transmitting / receiving terminal 103 described above, and the adjustment unit 13 corresponds to the control device 102 described above. The imaging unit 11 is a functional unit for generating the first video information described later. The imaging unit 11 captures video using various known methods and generates the first video information.

[0020] The acquisition unit 12 acquires video information. More specifically, the acquisition unit 12 acquires video information output from the imaging unit 11. The video information is information that indicates the target image, which is the image to be compressed among multiple images included in the video. For example, it is information that indicates one of multiple images (one frame) that are continuously input from the camera 101. The target image may be one image or multiple images. The video information is, for example, information generated by the camera 101 described above. The target image is, for example, encoded and then transmitted to the outside in real time by the transmission unit 14 described later. The generation of video information can be achieved by various known methods.

[0021] The acquisition unit 12 acquires first setting information from the relay device 20. The first setting information is information indicating at least one of the following: the amount of data of a target image (1 frame) included in the video, the resolution of the target image, and the frame rate of the target image. Resolution is an indicator that indicates the resolution of the target image, for example, an indicator that indicates the field of view of the target image. For example, resolutions include HD, FHD, and 4K. The higher the resolution, the more detailed the target image becomes and the larger the amount of data (bitrate) of the target image becomes. Frame rate is an indicator that indicates the number of frames displayed per second. The higher the frame rate, the smoother the video becomes and the larger the amount of data of the video becomes. The frame rate of the target image is the number of images displayed per second in a video containing the target image, for example, it indicates the number of frames when the frame interval between the timing when the target image is displayed and the timing when the immediately preceding consecutive image is displayed lasts for one second. For example, the frame rate of the target image is set by setting the time interval between the target image and the immediately preceding consecutive image.

[0022] An example of the process for generating the first setting information is described below. For example, the transmission unit 14, described later, links the imaging unit 11 (for example, a camera) and the adjustment unit 13 (for example, an encoder) with a web application or the like, and transmits (shares) the frame rate and resolution of the video captured by the imaging unit 11 to the relay device 20 via the web application or the like. The transmission unit 14 then transmits network status information, which indicates the state of the network between the transmitting terminal 10 and the relay device 20, or radio wave strength information, which relates to the radio wave strength of the transmitting terminal 10, to the relay device 20. The estimation unit 22 in the relay device 20, described later, generates network bandwidth information, which indicates the available bandwidth of the network between the transmitting terminal 10 and the relay device 20, based on the network status information or radio wave strength information. The estimation unit 22 generates first setting information such that the amount of video data does not exceed the available bandwidth indicated by the network bandwidth information. The transmission unit 25 transmits the first setting information to the transmitting terminal 10. Note that the network status information, radio wave strength information, and network bandwidth information are the same information as the status information, strength information, and bandwidth information described later. Furthermore, the process for generating each piece of information is the same as the process described later.

[0023] The adjustment unit 13 compresses the video information based on the first setting information. More specifically, the adjustment unit 13 adjusts the data size of the target image, the resolution of the target image, and the frame rate of the target image according to the first setting information. This allows the adjustment unit 13 to adjust the encoding bitrate so that the data size of the video including the target image does not exceed the available bandwidth indicated by the network bandwidth information. The adjustment unit 13 then compresses the target image using an encoder such as the transmission terminal 10 or a web application. The encoder is an encoder that compresses the target video using a lossy encoding method, and examples include encoders that support encoding methods such as H.265 / HEVC, H.264 / AVC, MPEG-4, MPEG-2, AV1 / VP10, and VP9.

[0024] The transmission unit 14 transmits video information representing the target image, compressed by the adjustment unit 13, to the relay device 20 via the network. More specifically, the transmission unit 14 packets the video information using a web application or the like (for example, the transmission unit 14 performs buffer control and queue control on the video information (for example, UDP (User Datagram Protocol) or TCP (Transmission Control Protocol), and also performs RTP transmission control such as RTSP (Real-Time Streaming Protocol), SRTP (Secure Real-Time Transport Protocol), and RTSPS (Real-Time Streaming Protocol Secure)), and sends the video information via the cellular network using a web application or the like. The packetized video information consists of an IP header, a TCP header or UDP header, an RTP header, and RTP data (video and audio).

[0025] The relay device 20 is configured as a functional component and includes an acquisition unit 21, an estimation unit 22, a classification unit 23, a generation unit 24, and a transmission unit 25. The acquisition unit 21, estimation unit 22, classification unit 23, generation unit 24, and transmission unit 25 correspond to the estimation server 201 and distribution server 202 described above.

[0026] The acquisition unit 21 acquires first video information. More specifically, the acquisition unit 21 acquires video information output from the transmission terminal 10. The acquisition unit 21 acquires first video information by decoding the video information. The first video information is information indicating a first target image, which is an image to be compressed among a plurality of images contained in the video, and is, for example, information indicating one of a plurality of consecutive images (one frame). The first target image may be one image or multiple images. The generation of the first video information is realized by various publicly known methods.

[0027] The acquisition unit 21 acquires multiple status information. The acquisition unit 21 acquires multiple status information periodically (for example, every second). The multiple status information is information indicating the status of each communication path P between the relay device 20 and each of the multiple terminals 30. Each status information is the round trip time (RTT) from the relay device 20 to the terminal 30. The round trip time is an indicator of the time it takes for a data packet to be sent from the source to the destination and then returned from the destination to the source. More specifically, the round trip time is the sum of the time from when the relay device 20 transmits the data packet until the terminal 30 receives the data packet, and the time from when the terminal 30 transmits an acknowledgment of the data packet to the relay device 20 until the relay device 20 receives it. For example, the acquisition unit 21 acquires distribution information (corresponding to the acknowledgment of the data packet) transmitted from the terminal 30 and generates status information based on the distribution information. For example, the acquisition unit 21 generates status information using various known methods based on the protocol being transmitted and received with the terminal 30 (for example, RTCP: Real-time Transport Control Protocol).

[0028] Each state information includes a first state information relating to the past state of communication path P, and a second state information relating to the state of communication path P even further in the past than that past. The first state information is the round-trip delay time from the relay device 20 to the terminal 30 in the past, and the second state information is the round-trip delay time even further in the past than that past. For example, "past" means the point in time immediately preceding the present, and "even further in the past" means a point in time before the point in time immediately preceding the present.

[0029] The acquisition unit 21 acquires multiple past bandwidth information regarding the available bandwidth of multiple past communication paths P. The available bandwidth of a communication path P is the bandwidth obtained by subtracting the bandwidth that cannot be used in the communication path P from the bandwidth of the communication path P. Bandwidth is an indicator of the amount of data that can be transferred through the communication path P. For example, the past bandwidth information includes information indicating the amount of data (bitrate) of images (frames) (hereinafter referred to as "past images") that the transmission unit 25 has encoded and transmitted via the communication path P in the past. The amount of data of the encoded past images is smaller than the available bandwidth of the past communication path P. For example, the past bandwidth information is the transmission throughput (TP: Throughput) in the relay device 20, and as one example, it is the amount of data of past images that were actually output from the generation unit 24 to the transmission unit 25 and transmitted to the terminal 30.

[0030] The acquisition unit 21 may acquire multiple strength information instead of multiple state information and multiple past bandwidth information. The multiple strength information is information about the radio wave strength at multiple terminals 30. Each strength information is, for example, information indicating RSRP (Reference Signal Received Power), information indicating RSSI (Received Signal Strength Indicator), or information indicating SINR (Signal-to-Interference-plus-Noise Ratio). Each strength information is generated at each of the multiple terminals 30 using various known methods and transmitted to the relay device 20.

[0031] The estimation unit 22 estimates multiple terminal bandwidth information, each indicating the available bandwidth of the communication path P, based on multiple state information. Terminal bandwidth information is information indicating the available bandwidth of the communication path P. More specifically, the estimation unit 22 estimates terminal bandwidth information based on first state information, second state information, and past bandwidth information. For example, as shown in Figure 3, the estimation unit 22 estimates terminal bandwidth information using a pre-set bandwidth estimation rule. The bandwidth estimation rule is, for example, a learning model that takes first state information, second state information, and past bandwidth information as input and outputs terminal bandwidth information. In this case, the estimation unit 22 pre-constructs a learning model by machine learning using first state information, second state information, past bandwidth information, and measured data of terminal bandwidth information as training data. The estimation unit 22 obtains terminal bandwidth information output from the learning model by inputting first state information, second state information, and past bandwidth information into the learning model.

[0032] Furthermore, for example, the bandwidth estimation rule is given by the formula shown in equation (1) below: Transmit TP(t+1) = F(RTT(t-1), RTT(t), Transmit TP(t-1)) + α ... (1) In equation (1) above, RTT(t) represents the first state information, RTT(t-1) represents the second state information, Transmit TP(t-1) represents past bandwidth information, α represents the correction value, and Transmit TP(t+1) represents the terminal bandwidth information. F is a formula that represents the bandwidth estimation rule, and is set by performing estimation based on the empirical rules of operators, etc., or based on a model learned using actual data, so that Transmit TP(t+1) is smaller than the measured value of the available bandwidth of the network. Note that the method of estimating the available bandwidth is not limited to the above.

[0033] The estimation unit 22 estimates terminal bandwidth information based on multiple state information, for example, at predetermined intervals. The predetermined interval is a pre-set time interval. As an example, the estimation unit 22 estimates terminal bandwidth information at time intervals smaller than several hundred milliseconds (e.g., 300 milliseconds). The estimation unit 22 estimates terminal bandwidth information every tens of milliseconds (e.g., 20 milliseconds, 33 milliseconds, and 66 milliseconds, etc.).

[0034] The estimation unit 22 may estimate multiple terminal bandwidth information based on multiple intensity information rather than multiple state information. More specifically, the estimation unit 22 may pre-acquire relational information showing the relationship between intensity information and terminal bandwidth information. The estimation unit 22 may extract terminal bandwidth information corresponding to each of the multiple intensity information by referring to the relational information based on the multiple intensity information. The method for generating terminal bandwidth information may be implemented by various publicly known methods. Furthermore, the process of estimating multiple terminal bandwidth information is executed periodically (for example, every second).

[0035] The classification unit 23 classifies the multiple terminals 30 into one of several groups based on the information of multiple terminal bandwidths. For example, each of the multiple groups corresponds to one of the multiple encoders E. More specifically, the multiple terminals 30 are classified into one of several groups based on the size of the available bandwidth of the communication path P between the relay device 20 and the terminals 30. For example, if the relay device 20 has five encoders E and 25 terminals 30 have a communication path P with the relay device 20, the five terminals 30 with the largest available bandwidth of the communication path P between the relay device 20 and the terminals 30 are grouped together. The next five terminals 30 with the next largest available bandwidth of the communication path P are grouped together. The same process is performed for the remaining 15 terminals 30, so that five terminals 30 are classified into each group.

[0036] Furthermore, for example, range information indicating the range of available bandwidth is pre-set for each of the multiple groups. The classification unit 23 classifies terminal 30 into the group corresponding to the range information if the range of available bandwidth indicated by the range information includes the available bandwidth indicated by the terminal bandwidth information relating to terminal 30. The classification unit 23 classifies terminal 30 into the group corresponding to the range information with the widest or narrowest range of available bandwidth if the available bandwidth indicated by the terminal bandwidth information relating to terminal 30 is not included in any of the ranges of available bandwidth indicated by each range information.

[0037] The classification unit 23 designates the terminal 30 corresponding to the terminal bandwidth information showing the smallest available bandwidth in each group as the representative terminal. The classification unit 23 periodically performs the above-mentioned grouping and representative terminal selection (for example, every minute).

[0038] The generation unit 24 acquires a plurality of group bandwidth information, each representing the available bandwidth corresponding to a plurality of groups. More specifically, the generation unit 24 generates group bandwidth information corresponding to one group based on at least one terminal bandwidth information corresponding to at least one terminal 30 classified into one of the plurality of groups from the plurality of terminal bandwidth information.

[0039] For example, the generation unit 24 determines the terminal bandwidth information with the smallest available bandwidth among the at least one terminal bandwidth information to be the group bandwidth information. As an example, the generation unit 24 generates terminal bandwidth information relating to a representative terminal for one group in the same way as the estimation unit 22, and uses the generated terminal bandwidth information as the group bandwidth information. The generation unit 24 performs the process of generating the group bandwidth information, for example, every tens of milliseconds. If the frequency at which the estimation unit 22 generates terminal bandwidth information is the same as the frequency at which the generation unit 24 generates group bandwidth information, the generation unit 24 obtains the terminal bandwidth information relating to the representative terminal from the estimation unit 22 and uses the obtained terminal bandwidth information as the group bandwidth information.

[0040] Furthermore, for example, the generation unit 24 may acquire a plurality of pre-set group bandwidth information. More specifically, if range information indicating the range of available bandwidth is pre-set for each of the plurality of groups, the generation unit 24 may use the lower limit of the range of available bandwidth indicated by the range information as the group bandwidth information for the group corresponding to the range information.

[0041] The generation unit 24 generates a plurality of second video information, each representing a plurality of second target images corresponding to a plurality of groups, based on the plurality of group bandwidth information and the first video information. More specifically, the generation unit 24 generates the second video information such that the data amount of the second target image corresponding to one of the plurality of groups does not exceed the available bandwidth indicated by the group bandwidth information corresponding to that group. More specifically, the generation unit 24 performs correction of the resolution and frame rate of the first target image using image extension function and frame insertion function, etc. (At this time, the generation unit 24 may process the first target image by dynamic range correction and color correction, etc.). The generation unit 24 generates a plurality of second video information, each representing a plurality of second target images, by encoding the first target image with the encoder E for each group. In this case, the generation unit 24 recompresses the first target image to generate the second target image so that the data amount of the second target image does not exceed the available bandwidth indicated by the group bandwidth information.

[0042] For example, the generation unit 24 generates second setting information indicating the data amount (bitrate), resolution, and frame rate of the second target image of the second target image, such that the data amount of the second target image (1 frame) to be encoded next does not exceed the available bandwidth indicated by the group bandwidth information. Based on this second setting information, the generation unit 24 recompresses the first target image to generate the second target image. At this time, the generation unit 24 determines (instructs) the data amount (target bitrate) of the second target image (1 frame) to be encoded next so that the data amount of the second target image (1 frame) is comfortably below the available bandwidth indicated by the group bandwidth information.

[0043] Furthermore, if the available bandwidth (transmission throughput) indicated by the group bandwidth information is small, the generation unit 24 may generate second setting information based on the amount of data (average amount of data) for each resolution of the second target image (1 frame) indicated by the group bandwidth information and the second video information. In this case, the generation unit 24 changes the resolution of the second target image (1 frame) to be encoded next based on the amount of data for each resolution of the second target image (1 frame) so that the amount of data of the next target image to be encoded does not exceed the available bandwidth indicated by the group bandwidth information.

[0044] Further, when the available bandwidth (transmission throughput) indicated by the group bandwidth information is small, the generation unit 24 generates second setting information based on the group bandwidth information and the operation information of the camera 101. In this case, when determining that there is no operation on the camera 101 based on the operation information of the camera 101, the generation unit 24 may reduce the frame rate of the next second target image such that the data amount of the next second target image (one frame) to be encoded does not exceed the available bandwidth indicated by the group bandwidth information. As a result, when there is no operation of the camera 101, the moving image captured by the camera 101 is substantially a still image, so even if the frame rate is reduced for compression, the influence on the decoding process of the moving image including the second target image is small. Accordingly, the generation unit 24 can reduce the encoding bit rate while maintaining the resolution of the moving image and suppressing the influence on the decoding process of the moving image.

[0045] The transmission unit 25 transmits, to each of the plurality of terminals 30, the second moving image information corresponding to the group into which each of the plurality of terminals 30 is classified, among the plurality of pieces of second moving image information. More specifically, the transmission unit 25 packetizes each piece of the second moving image information (for example, the transmission unit 25 performs buffer control and queue control on the second moving image information (as an example, executes UDP or TCP, and executes RTP transmission control such as RTSP, SRTP, and RTSPS)), and transmits the packetized second moving image information to the terminals 30 classified into the group corresponding to the second moving image information. The packetized second moving image information is composed of an IP header, a TCP header or a UDP header, an RTP header, and RTP data (moving image and audio).

[0046] As described above, the relay device 20 (for example, the distribution server 202) holds a plurality of encoders E. The classification unit 23 (for example, the estimation server 201) classifies the plurality of terminals 30 into any of the groups of the same number as the number of encoders, based on the terminal bandwidth information related to the plurality of terminals 30. The generation unit 24 (for example, the distribution server 202) compresses the first moving image information based on the smallest available bandwidth of each group to generate the second moving image information. Accordingly, the relay device 20 can distribute, to each terminal 30, the second moving image information indicating a moving image adapted to the network condition of the viewer.

[0047] The terminal 30 includes, as functional components, an acquisition unit 31, a decoding unit 32, a playback unit 33, and a transmission unit 34. The acquisition unit 31 acquires second moving image information from the relay device 20.

[0048] The decoding unit 32 decodes the second moving image information output from the acquisition unit 31. More specifically, the decoding unit 32 converts the second moving image information output from the acquisition unit 31 into first moving image information before being compressed by the generation unit 24 of the relay device 20. That is, the decoding unit 32 decodes the second moving image information with an irreversible encoding scheme, and restores the second moving image information to the first moving image information before being compressed by the adjustment unit 13, based on the data amount of the first target image included in the second moving image information, the resolution of the first target image, and the frame rate information of the first target image. The decoding method for the second moving image information can be implemented by various known techniques.

[0049] The playback unit 33 plays back the restored first moving image information. Accordingly, the user can view the first target image indicated by the first moving image information, and thus can watch the moving image composed of the plurality of first target images.

[0050] The transmission unit 34 transmits distribution information (acknowledgment for data packets) to the relay device 20 so that the acquisition unit 21 of the relay device 20 can acquire state information. The transmission unit 34 may periodically transmit a plurality of pieces of intensity information to the relay device 20 (for example, every one second).

[0051] FIG. 4 is a flowchart illustrating an example of processing of a transmission method executed by the relay device 20 (transmission device). With reference to FIG. 4, an example of the processing of the transmission method executed by the relay device 20 will be described.

[0052] First, the relay device 20 acquires first video information and multiple state information or multiple intensity information (steps S1 and S2: acquisition step). Next, the relay device 20 estimates multiple terminal bandwidth information based on the multiple state information or multiple intensity information (step S3: estimation step). Next, the relay device 20 classifies multiple terminals 30 into one of multiple groups (step S4: classification step). Next, the relay device 20 acquires multiple group bandwidth information and generates multiple second video information based on the multiple group bandwidth information and the first video information (steps S5 and S6: generation step). Next, the relay device 20 transmits the multiple second video information (step S7: transmission step).

[0053] Next, the effects and benefits of the relay device 20 and transmission method according to this embodiment will be described.

[0054] According to one embodiment of the present disclosure, the relay device 20 acquires a plurality of group bandwidth information indicating the available bandwidth corresponding to each of a plurality of groups, generates a plurality of second video information indicating a plurality of second target images corresponding to each of the plurality of groups based on the plurality of group bandwidth information and first video information, and transmits to each of the plurality of terminals the second video information corresponding to the group to which each of the plurality of terminals is classified from among the plurality of second video information. With this configuration, the plurality of second video information to be transmitted to each of the plurality of terminals 30 is generated taking into account the available bandwidth of the communication path P between the transmitting device and each of the plurality of terminals. This makes it possible to smoothly play the video on each of the plurality of terminals 30. Furthermore, with the above configuration, it is possible to reduce the processing load on the relay device 20 compared to the case in which second video information is generated for each terminal 30. As a result, it is possible to smoothly play the video on the receiving device while reducing the processing load on the transmitting device.

[0055] The generation unit 24 generates group bandwidth information corresponding to one group based on at least one terminal bandwidth information corresponding to at least one terminal 30 classified into one of the multiple groups from among the terminal bandwidth information. In this case, the group bandwidth information is generated considering the available bandwidth of each communication path P between at least one terminal 30 classified into one group and the relay device 20. This makes it possible to smoothly play video on each of the multiple terminals 30.

[0056] The generation unit 24 determines the group bandwidth information to be the terminal bandwidth information with the smallest available bandwidth from among the terminal bandwidth information corresponding to at least one terminal 30 classified into one of the multiple groups. In this case, the available bandwidth indicated by the group bandwidth information can be set to be less than or equal to the available bandwidth of each communication path P between all terminals 30 classified into one group and the relay device 20. This makes it possible to more reliably play videos smoothly on each of the multiple terminals 30.

[0057] The generation unit 24 acquires a set number of pre-configured group bandwidth information. In this case, compared to the case where the terminal bandwidth information with the smallest available bandwidth is determined as the group bandwidth information, the number of group bandwidth information can be acquired more quickly, thus enabling the generation of the second video information more rapidly.

[0058] The generation unit 24 generates the second video information such that the available bandwidth indicated by the group bandwidth information corresponding to one of the multiple groups does not exceed the data amount of the second target image corresponding to that one group. In this case, it becomes possible to reliably play the video smoothly on each of the multiple terminals 30.

[0059] In the above embodiment, multiple groups were pre-configured, but are not limited to this. For example, the classification unit 23 may configure multiple groups based on multiple terminal bandwidth information.

[0060] For example, the classification unit 23 may first select five available bandwidths from the available bandwidths of multiple communication paths P indicated by multiple terminal bandwidth information (for example, classifying the available bandwidths of multiple communication paths P into five levels and selecting the smallest available bandwidth at each level). Based on the five selected available bandwidths, the classification unit 23 may determine the data amount (bitrate) of the first target image when compressing the first target image, the resolution of the first target image, and the frame rate of the first target image (the number of images displayed per second in a video containing the first target image; for example, the number of images when the frame interval between the timing at which the first target image is displayed and the timing at which the immediately preceding consecutive image is displayed in a video containing the first target image lasts for one second. For example, the frame rate of the first target image is set by setting the time interval between the target image and the immediately preceding consecutive image). The classification unit 23 may periodically perform the above-described determination process (for example, every minute) and update the instructions to the encoder E.

[0061] The classification unit 23 may acquire multiple terminal bandwidth information estimated periodically (for example, every 20 milliseconds) by the estimation unit 22. The classification unit 23 may classify the terminal 30 corresponding to the terminal bandwidth information into an encoder E that generates a second target image with a data volume that does not exceed the available bandwidth indicated by the terminal bandwidth information. That is, the classification unit 23 classifies the terminal 30 into one of multiple groups (multiple encoders E) based on the multiple terminal bandwidth information. In this case, if the available bandwidth indicated by the terminal bandwidth information is the smallest, and the estimated value of that available bandwidth is smaller than the five available bandwidths selected by the classification unit 23, the classification unit 23 determines the data volume (bitrate), resolution, and frame rate according to the estimated value of the available bandwidth, and updates the instruction to the encoder E that is set to have the smallest available bandwidth.

[0062] In this way, the multiple encoders E compress the first target image with a preset data amount (bitrate), resolution, and frame rate. The classification unit 23 associates the terminal 30 with an encoder E that outputs data that does not exceed the terminal bandwidth information related to the terminal 30. The transmission unit 25 transmits the compressed data generated by the encoder E associated with the terminal 30 to the terminal 30. Therefore, the transmission unit 25 selects compressed data from among the compressed data generated by the multiple encoders E that does not exceed the available bandwidth indicated by the terminal bandwidth information related to the terminal 30 and delivers it to the terminal 30.

[0063] Furthermore, in the above embodiment, video compression and decoding were performed using encoders and decoders provided in the transmitting terminal 10 and the relay device 20, but they may also be performed using encoders and decoders on another server that is connected to the transmitting terminal 10 and the relay device 20 via a web application or the like.

[0064] The transmitting device and transmitting method of the present disclosure have the following configuration: [1] A transmitting device for transmitting video to a plurality of terminals, comprising: an acquisition unit that acquires first video information indicating a first target image which is an image to be compressed from among a plurality of images included in the video, a plurality of state information indicating the state of each communication path between the transmitting device and each of the plurality of terminals, or a plurality of intensity information relating to the radio wave intensity at the plurality of terminals; an estimation unit that estimates a plurality of terminal bandwidth information indicating the available bandwidth of each communication path based on the plurality of state information or the plurality of intensity information; a classification unit that classifies the plurality of terminals into one of a plurality of groups based on the plurality of terminal bandwidth information; a generation unit that acquires a plurality of group bandwidth information indicating each available bandwidth corresponding to each of the plurality of groups, and generates a plurality of second video information indicating a plurality of second target images corresponding to each of the plurality of groups based on the plurality of group bandwidth information and the first video information; and a transmitting unit that transmits to each of the plurality of terminals the second video information corresponding to the group to which each of the plurality of terminals is classified from among the plurality of second video information. [2] The transmission device according to [1], wherein the generation unit generates group bandwidth information corresponding to one group based on at least one terminal bandwidth piece of information corresponding to at least one terminal classified into one of the groups from the plurality of terminal bandwidth information. [3] The transmission device according to [2], wherein the generation unit determines the terminal bandwidth information with the smallest available bandwidth from the at least one terminal bandwidth piece of information as the group bandwidth information. [4] The transmission device according to any one of [1] to [3], wherein the generation unit acquires the plurality of pre-set group bandwidth information. [5] The transmission device according to any one of [1] to [4], wherein the generation unit generates the second video information such that the available bandwidth indicated by the group bandwidth information corresponding to one of the plurality of groups does not exceed the amount of data of the second target image corresponding to the one group.[6] A transmission method performed by a transmitting device, comprising: an acquisition step in which the transmitting device acquires first video information indicating a first target image which is an image to be compressed among a plurality of images included in a video, a plurality of state information indicating the state of each communication path between the transmitting device and each of the plurality of terminals, or a plurality of intensity information relating to the radio wave intensity at the plurality of terminals; an estimation step in which the transmitting device estimates a plurality of terminal bandwidth information indicating the available bandwidth of each communication path based on the plurality of state information or the plurality of intensity information; a classification step in which the transmitting device classifies the plurality of terminals into one of a plurality of groups based on the plurality of terminal bandwidth information; a generation step in which the transmitting device acquires a plurality of group bandwidth information indicating each available bandwidth corresponding to each of the plurality of groups, and generates a plurality of second video information indicating a plurality of second target images corresponding to each of the plurality of groups based on the plurality of group bandwidth information and the first video information; and a transmission step in which the transmitting device transmits to each of the plurality of terminals the second video information corresponding to the group in which each of the plurality of terminals is classified from among the plurality of second video information.

[0065] Next, the hardware configuration of the relay device 20 described above will be explained with reference to Figure 5. Physically, the relay device 20 may be configured as a computer device including a processor 1001, memory 1002, storage 1003, communication device 1004, input device 1005, output device 1006, bus 1007, etc.

[0066] In the following explanation, the term "device" can be replaced with "circuit," "device," "unit," etc. The hardware configuration of the relay device 20 may include one or more of the devices shown in the figure, or it may be configured to omit some of the devices.

[0067] Each function in the relay device 20 is realized by loading predetermined software (programs) onto hardware such as the processor 1001 and memory 1002, causing the processor 1001 to perform calculations and control communication by the communication device 1004 and the reading and / or writing of data in the memory 1002 and storage 1003.

[0068] The processor 1001 controls the entire computer, for example, by running the operating system. The processor 1001 may consist of a central processing unit (CPU) that includes interfaces with peripheral devices, control devices, arithmetic units, registers, etc. For example, control functions such as the estimation unit 22 may be implemented in the processor 1001.

[0069] Furthermore, the processor 1001 reads programs (program code), software modules, and data from the storage 1003 and / or communication device 1004 into the memory 1002, and executes various processes accordingly. The program used is one that causes the computer to execute at least a part of the operations described in the above embodiment.

[0070] For example, control functions such as the estimation unit 22 may be stored in memory 1002 and implemented by a control program running on processor 1001, and other functional blocks may be implemented similarly. Although the above-described processes have been explained as being executed by one processor 1001, they may be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented on one or more chips. The program may also be transmitted from a network via a telecommunications line.

[0071] The memory 1002 is a computer-readable recording medium and may consist of at least one of the following: ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), RAM (Random Access Memory), etc. The memory 1002 may also be called a register, cache, main memory, etc. The memory 1002 can store executable programs (program code), software modules, etc., for carrying out the transmission method according to one embodiment of the present invention.

[0072] The storage 1003 is a computer-readable recording medium and may consist of at least one of the following: an optical disc such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital multipurpose disk, a Blu-ray® disc), a smart card, flash memory (e.g., a card, a stick, a key drive), a floppy® disk, a magnetic strip, etc. The storage 1003 may also be called an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, server, or other suitable medium including memory 1002 and / or storage 1003.

[0073] The communication device 1004 is hardware (transceiver / receiver device) for communicating between computers via a wired and / or wireless network, and is also referred to as a network device, network controller, network card, communication module, etc.

[0074] The input device 1005 is an input device that accepts input from an external source (e.g., a keyboard, mouse, microphone, switch, button, sensor, etc.). The output device 1006 is an output device that outputs to an external source (e.g., a display, speaker, LED lamp, etc.). The input device 1005 and the output device 1006 may be configured as an integrated unit (e.g., a touch panel).

[0075] Furthermore, each device, such as the processor 1001 and the memory 1002, is connected by a bus 1007 for communicating information. The bus 1007 may consist of a single bus, or different buses may be used for communication between devices.

[0076] Furthermore, the relay device 20 may be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array), and some or all of each functional block may be realized by such hardware. For example, the processor 1001 may be implemented using at least one of these hardware components.

[0077] Although this embodiment has been described in detail above, it will be clear to those skilled in the art that this embodiment is not limited to the embodiments described herein. This embodiment can be implemented as a modified and altered form without departing from the spirit and scope of the invention as defined by the claims. Therefore, the description herein is for illustrative purposes only and is not intended to be restrictive in any way to this embodiment.

[0078] Each aspect / embodiment described herein may be applied to systems utilizing LTE (Long Term Evolution), LTE-A (LTE-Advanced), SUPER 3G, IMT-Advanced, 4G, 5G, FRA (Future Radio Access), W-CDMA®, GSM®, CDMA2000, UMB (Ultra Mobile Broad-band), IEEE 802.11 (Wi-Fi), IEEE 802.16 (WiMAX), IEEE 802.20, UWB (Ultra-Wide Band), Bluetooth®, and other appropriate systems, and / or next-generation systems extended based thereon.

[0079] The processing procedures, sequences, flowcharts, etc., of each aspect / embodiment described herein may be reordered, provided they are consistent with each other. For example, the methods described herein present the elements of various steps in an exemplary order and are not limited to that specific order.

[0080] Input and output information may be stored in a specific location (e.g., memory) or managed in a management table. Input and output information may be overwritten, updated, or appended to. Output information may be deleted. Input information may be transmitted to other devices.

[0081] The determination may be made by a value represented by one bit (0 or 1), by a boolean value (true or false), or by a numerical comparison (for example, a comparison with a predetermined value).

[0082] Each aspect / embodiment described herein may be used individually, in combination, or switched between during implementation. Furthermore, notification of predetermined information (e.g., notification that "X is") is not limited to explicit notification, but may also be implicit (e.g., by not providing such notification).

[0083] Software should be broadly interpreted to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, and so on, whether they are called software, firmware, middleware, microcode, hardware description languages, or by any other name.

[0084] Furthermore, software, instructions, etc., may be transmitted and received via a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies such as coaxial cable, fiber optic cable, twisted pair, and digital subscriber lines (DSL) and / or wireless technologies such as infrared, radio, and microwave, these wired and / or wireless technologies are included in the definition of a transmission medium.

[0085] The information, signals, etc. described herein may be represented using any one of various different techniques. For example, the data, instructions, commands, information, signals, bits, symbols, chips, etc., which may be referred to throughout the above description, may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0086] In addition, terms used herein and / or terms necessary for understanding this specification may be replaced with terms having the same or similar meanings.

[0087] Furthermore, the information, parameters, etc., described herein may be expressed as absolute values, relative values ​​from a given value, or as corresponding other information.

[0088] Communication terminals may also be referred to by those skilled in the art as mobile communication terminals, subscriber stations, mobile units, subscriber units, wireless units, remote units, mobile devices, wireless devices, wireless communication devices, remote devices, mobile subscriber stations, access terminals, mobile terminals, wireless terminals, remote terminals, handsets, user agents, mobile clients, clients, or several other appropriate terms.

[0089] As used herein, the phrase "based on" does not mean "based solely on" unless otherwise specified. In other words, the phrase "based on" means both "based solely on" and "based at least on."

[0090] Where the terms “first,” “second,” etc., are used herein, no reference to those elements shall generally limit the quantity or order of those elements. These terms may be used herein as a convenient way to distinguish between two or more elements. Accordingly, references to the first and second elements shall not imply that only two elements may be employed therein, or that the first element must precede the second element in any way.

[0091] To the extent that “include,” “including,” and their variations are used herein or in the claims, these terms are intended to be inclusive, as is the term “comprising.” Furthermore, the term “or” as used herein or in the claims is not intended to be exclusive OR.

[0092] In this specification, unless the context or technically it is clear that only one device exists, the term also includes multiple devices.

[0093] Throughout this disclosure, unless the context clearly indicates a singular number, the terms shall include plural ones.

[0094] 20... Relay device (transmitter), 21... Acquisition unit, 22... Estimation unit, 23... Classification unit, 24... Generation unit, 25... Transmitter unit, 30... Terminal, 1001... Processor, 1002... Memory, 1003... Storage, 1004... Communication device, 1005... Input device, 1006... Output device, 1007... Bus.

Claims

1. A transmitting device for transmitting video to multiple terminals, comprising: an acquisition unit that acquires first video information indicating a first target image which is an image to be compressed from among a plurality of images included in the video; a plurality of state information indicating the state of each communication path between the transmitting device and each of the plurality of terminals, or a plurality of intensity information relating to the radio wave intensity at the plurality of terminals; an estimation unit that estimates a plurality of terminal bandwidth information indicating the available bandwidth of each communication path based on the plurality of state information or the plurality of intensity information; a classification unit that classifies the plurality of terminals into one of a plurality of groups based on the plurality of terminal bandwidth information; a generation unit that acquires a plurality of group bandwidth information indicating each available bandwidth corresponding to each of the plurality of groups, and generates a plurality of second video information indicating a plurality of second target images corresponding to each of the plurality of groups based on the plurality of group bandwidth information and the first video information; and a transmitting unit that transmits to each of the plurality of terminals the second video information corresponding to the group to which each of the plurality of terminals is classified from among the plurality of second video information.

2. The transmission device according to claim 1, wherein the generation unit generates group bandwidth information corresponding to one group based on at least one terminal bandwidth information corresponding to at least one terminal classified into one of the groups from the plurality of terminal bandwidth information.

3. The transmission device according to claim 2, wherein the generation unit determines the terminal bandwidth information with the smallest available bandwidth among the at least one terminal bandwidth information to be the group bandwidth information.

4. The transmission device according to claim 1, wherein the generation unit acquires the plurality of pre-set group bandwidth information.

5. The transmission device according to claim 1, wherein the generation unit generates the second video information such that the amount of data of the second target image corresponding to one of the plurality of groups does not exceed the available bandwidth indicated by the group bandwidth information corresponding to one of the plurality of groups.

6. A transmission method performed by a transmitting device, comprising: an acquisition step in which the transmitting device acquires first video information indicating a first target image which is an image to be compressed among a plurality of images included in a video, a plurality of state information indicating the state of each communication path between the transmitting device and each of the plurality of terminals, or a plurality of intensity information relating to the radio wave intensity at the plurality of terminals; an estimation step in which the transmitting device estimates a plurality of terminal bandwidth information indicating the available bandwidth of each communication path based on the plurality of state information or the plurality of intensity information; a classification step in which the transmitting device classifies the plurality of terminals into one of a plurality of groups based on the plurality of terminal bandwidth information; a generation step in which the transmitting device acquires a plurality of group bandwidth information indicating each available bandwidth corresponding to each of the plurality of groups, and generates a plurality of second video information indicating a plurality of second target images corresponding to each of the plurality of groups based on the plurality of group bandwidth information and the first video information; and a transmission step in which the transmitting device transmits to each of the plurality of terminals the second video information corresponding to the group to which each of the plurality of terminals is classified from among the plurality of second video information.

Images