Transmission system and transmission method
Patent Information
- Application Number
- PCT/JP2026/005318
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-04-14
- Filing Date
- 2026-02-13
- Publication Date
- 2026-10-22
Smart Images
Figure JP2026005318_22102026_PF_FP_ABST
Abstract
Description
Transmission System and Transmission Method
[0001] One aspect of the present disclosure relates to a transmission system and a transmission method.
[0002] Conventionally, a terminal device that transmits video data to a server device via a network is known (for example, see Patent Document 1). For example, in the terminal device described in Patent Document 1, resolution adjustment of image data, frame rate adjustment, and the like are performed on the video data based on bandwidth information associated with information indicating a network state, so that the video data is transmitted to the server device in a state where the bandwidth used by the terminal device is adjusted to a target bandwidth.
[0003] Japanese Unexamined Patent Publication No. 2014-175959
[0004] However, when the above-described terminal device continues to transmit moving images via a network, the available bandwidth (transmission throughput) of the network changes, so that even if adjustment is performed so that the bandwidth used by the terminal device becomes the target bandwidth, the target bandwidth deviates from the actual available bandwidth of the network. In this case, the moving image cannot be smoothly reproduced at the transmission destination device, and there is a risk that the video data will be intermittent during reproduction. Accordingly, it is desired to enable smooth reproduction of the moving image at the transmission destination device.
[0005] One aspect of the present disclosure has been made in view of the above circumstances, and an object thereof is to provide a transmission system and a transmission method that enable smooth reproduction of a moving image at a terminal that is the transmission destination of the moving image.
[0006] A transmission system according to one aspect of this disclosure is a transmission system for transmitting video over a network, comprising: an acquisition unit that acquires video information indicating a target image which is to be compressed among a plurality of images contained in the video, status information relating to the state of the network, and past bandwidth information relating to the available bandwidth of the network in the past; an estimation unit that estimates future bandwidth information indicating the available bandwidth of the network in the future based on the status information and past bandwidth information; an adjustment unit that adjusts setting information indicating at least one of the resolution of the target image and the frame rate of the target image based on the future bandwidth information and video information, and compresses the video information based on the setting information; and a transmission unit that transmits the video information over a network.
[0007] In one form of the present disclosure, a transmission system estimates future network bandwidth information based on status information indicating the network state and past bandwidth information regarding past network bandwidth availability. Based on the future bandwidth information, it adjusts setting information indicating at least one of the resolution of the target image and the frame rate of the target image, and compresses the video information based on the setting information. With this configuration, even if the network bandwidth availability changes, the new bandwidth availability can be estimated with high accuracy. As a result, by compressing the video information according to the setting information based on the future bandwidth information indicating the new bandwidth availability, it becomes possible to smoothly play the video at the receiving device of the video information.
[0008] A transmission method relating to one aspect of the present disclosure is a transmission method performed by a transmission system, comprising: an acquisition step in which the transmission system acquires video information indicating a target image which is to be compressed among a plurality of images contained in a video, state information relating to the state of the network, and past bandwidth information relating to the past available bandwidth of the network; an estimation step in which the transmission system estimates future bandwidth information relating to the future available bandwidth of the network based on the state information and the past bandwidth information; an adjustment step in which the transmission system adjusts setting information indicating at least one of the resolution of the target image and the frame rate of the target image based on the future bandwidth information and the video information, and compresses the video information based on the setting information; and a transmission step in which the transmission system transmits the video information over the network.
[0009] According to one form of transmission system and transmission method described herein, it becomes possible to smoothly play the video on the receiving device.
[0010] Figure 1 is a schematic diagram showing an overview of an information processing system equipped with a transmitting device according to one embodiment. Figure 2 is a block diagram showing the functional configuration of the transmitting device and relay device shown in Figure 1. Figure 3 is a diagram illustrating the process of estimating future bandwidth information. Figure 4 is a flowchart showing an example of the processing of a transmission method executed by the transmitting device according to one embodiment. Figure 5 is a flowchart showing an example of the processing of a transmission method executed by a modified transmitting device. Figure 6 is a diagram showing the hardware configuration of the transmitting device according to the embodiment.
[0011] Embodiments of this disclosure will be described in detail below with reference to the drawings. In each drawing, the same or corresponding parts are denoted by the same reference numerals, and redundant descriptions are omitted.
[0012] Figure 1 is a schematic diagram showing an overview of an information processing system 1 equipped with a transmitting device 10 (transmission system) according to one embodiment. The information processing system 1 comprises a transmitting device 10, a relay device 20, and a control device 30. The information processing system 1 is a system for transmitting video to multiple destination terminals (not shown). The transmitting device 10 transmits video to the relay device 20. The relay device 20 adjusts the video received from the transmitting device 10 and outputs it to the control device 30. The control device 30 edits the output video and transmits the edited video to multiple destination terminals (not shown).
[0013] For example, information processing system 1 is a system used by broadcasters such as television stations. Transmitter 10 is a device installed at the relay site and is a transmitter for transmitting video over a network. Relay device 20 is a device installed at the broadcasting station, for example, a sub-control room in the broadcasting station. Control device 30 is a device installed at the broadcasting station, for example, a device located in a studio within the broadcasting station.
[0014] The transmitting device 10 includes a camera 101, a transmitting / receiving terminal 102, a control device 103, and an encoder 104. The camera 101 is a device for capturing video at the relay site. The transmitting / receiving terminal 102 is, for example, a terminal or router capable of 5G communication. The control device 103 is a device for controlling the encoder 104. The encoder 104 is a device for compressing the video signal from the camera 101.
[0015] Camera 101 outputs the video it captures and generates at the relay site to encoder 104. Transmitting / receiving terminal 102 outputs information received from external devices such as relay device 20 and control device 30 to control device 103. Control device 103 determines the setting information for encoder 104 based on the information output from transmitting / receiving terminal 102. Based on the setting information output from control device 103, encoder 104 compresses the video signal output from camera 101 (for example, by adjusting the resolution and frame rate, etc.) and outputs it to transmitting / receiving terminal 102. Transmitting / receiving terminal 102 transmits the video output from encoder 104 to relay device 20, for example, via a 5G network.
[0016] The relay device 20 includes a transmitting / receiving terminal 201 and a decoder 202. The transmitting / receiving terminal 201 is, for example, a 5G communication-capable terminal or router. The transmitting / receiving terminal 201 outputs the video signal received from the transmitting device 10 to the decoder 202. The decoder 202 decodes the output video signal back into the original video signal and outputs it to the control device 30.
[0017] Information processing system 1 enables broadcasters, such as broadcasting stations, to receive and play back video in real time. Furthermore, by using a 5G network for communication, high-speed and low-latency video transmission and control can be achieved. The functional configurations of the transmitting device 10, relay device 20, and control device 30 will be described in detail below.
[0018] Figure 2 is a block diagram showing the functional configuration of the transmitting device 10 and the relay device 20. The transmitting device 10 is composed of the following functional components: an acquisition unit 11, an estimation unit 12, an adjustment unit 13, a transmitting unit 14, an imaging unit 15, and a control unit 16. The functions of each functional unit of the transmitting device 10 will be described in detail below. The acquisition unit 11 and the transmitting unit 14 correspond to the transmitting / receiving terminal 102 described above, the estimation unit 12 corresponds to the control device 103 described above, the adjustment unit 13 corresponds to the encoder 104 described above, and the imaging unit 15 and the control unit 16 correspond to the camera 101 described above.
[0019] The acquisition unit 11 acquires video information. More specifically, the acquisition unit 11 acquires video information output from the imaging unit 15. The video information is information that indicates the target image that is to be compressed among multiple images included in the video, for example, information that indicates one of multiple images (one frame) that are continuously input from the camera 101. The target image may be one image or multiple images. The video information is information generated by a camera installed at a relay station (for example, the camera 101 mentioned above). This video information is transmitted to the outside in real time by the transmission unit 14, which will be described later. The generation of video information is realized by various known methods.
[0020] The acquisition unit 11 acquires status information related to the network status. The network is the network between the transmitting device 10 (shooting location) and the device to which the video is transmitted (for example, the relay device 20 (relay point)). The status information is the round trip time (RTT) from the transmitting device 10 (the device that transmits the video information) to a device outside of the transmitting device 10 (for example, the device to which the video information is transmitted, which is the relay device 20). The round trip time is an indicator that shows the time it takes for a data packet to be sent from the source to the receiver and then returned from the receiver to the source. More specifically, the round trip time is the sum of the length of time from when the transmitting device 10 transmits the data packet until the relay device 20 receives the data packet, and the length of time from when the relay device 20 sends an acknowledgment of the data packet to the transmitting device 10 until the transmitting device 10 receives it. For example, the acquisition unit 11 acquires the distribution information (corresponding to the acknowledgment of the data packet) transmitted from the relay device 20 and generates status information based on the distribution information. For example, the acquisition unit 11 generates status information using various known methods based on the protocol (for example, RTCP: Real-time Transport Control Protocol) being transmitted and received with the relay device 20.
[0021] The status information includes first status information relating to the past state of the network and second status information relating to the network state even further in the past than that past. The first status information is the round-trip delay time from the transmitting device 10 to a device outside of the transmitting device 10 in the past, and the second status information is the round-trip delay time even further in the past than that past. For example, "past" means the point in time immediately preceding the present, and "even further in the past than that past" means a point in time before the point in time immediately preceding the present.
[0022] The acquisition unit 11 acquires past bandwidth information regarding the available bandwidth of the network in the past, for example, from the transmission unit 14. The available bandwidth of the network is the bandwidth of the network minus the bandwidth that cannot be used in the network. Bandwidth is an indicator of the amount of data that the network can transfer. For example, past bandwidth information includes information indicating the bitrate of images (frames) (hereinafter referred to as "past images") that the transmission unit 14 has encoded and transmitted in the past. The amount of data (bitrate) of the encoded past images is smaller than the available bandwidth of the network in the past. For example, past bandwidth information is the transmission throughput (TP) of the transmission device 10, and as an example, it is the amount of video data that was actually output from the adjustment unit 13 to the transmission unit 14 and transmitted to the relay device 20. Past images (frames) are images that were transmitted in the past than the target image described above. For example, a past image is one of several images that make up a single video, which was encoded and transmitted in the past than the target image described above. Note that a past image may be one image or multiple images.
[0023] The past bandwidth information may further include information indicating at least one of the resolution of the encoded past image and the frame rate of the past image. Frame rate generally refers to the number of images (frames) displayed per second. The frame rate of a past image refers to the number of images displayed per second in a video containing past images. For example, it refers to the number of images when the frame interval between the timing at which a past image is displayed and the timing at which the immediately preceding image is displayed in a video containing past images lasts for one second. As an example, the frame rate of a past image is set by setting a time interval between a past image and the immediately preceding image that follows it. In this case, the acquisition unit 11 may acquire the setting information adjusted by the adjustment unit 13 from the adjustment unit 13 and use it as past bandwidth information. Furthermore, the method for generating past bandwidth information can be implemented by various known methods.
[0024] The estimation unit 12 estimates future bandwidth information based on state information and past bandwidth information. Future bandwidth information is information that indicates the available bandwidth of the network in the future. The future can be any time after the time when the bandwidth information is estimated, for example, immediately after that time. More specifically, the estimation unit 12 estimates future bandwidth information based on first state information, second state information and past bandwidth information. For example, as shown in Figure 3, the estimation unit 12 estimates future bandwidth information using a pre-set bandwidth estimation rule. The bandwidth estimation rule is, for example, a learning model that takes first state information, second state information and past bandwidth information as input and outputs future bandwidth information. In this case, the estimation unit 12 pre-constructs a learning model by machine learning using first state information, second state information and past bandwidth information and measured data of future bandwidth information as training data. The estimation unit 12 obtains the future bandwidth information output from the learning model by inputting first state information, second state information and past bandwidth information into the learning model.
[0025] Furthermore, for example, the bandwidth estimation rule is given by the formula shown in equation (1) below: Transmit TP(t+1) = F(RTT(t-1), RTT(t), Transmit TP(t-1)) + α ... (1) In equation (1) above, RTT(t) represents the first state information, RTT(t-1) represents the second state information, Transmit TP(t-1) represents past bandwidth information, α represents a correction value, and Transmit TP(t+1) represents future bandwidth information. F is a formula that represents the bandwidth estimation rule, and is set by performing estimation based on the empirical rules of operators, etc., or based on a model learned using actual data, so that Transmit TP(t+1) is smaller than the measured value of the available bandwidth of the network. Note that the method of estimating the available bandwidth is not limited to the above.
[0026] The estimation unit 12 estimates future bandwidth information based on the status information and past bandwidth information, for example, at predetermined intervals. The predetermined interval is a pre-set time interval. As an example, the estimation unit 12 estimates future bandwidth information at time intervals smaller than several hundred milliseconds (e.g., 300 milliseconds). The estimation unit 12 estimates future bandwidth information every tens of milliseconds (e.g., 20 milliseconds, 33 milliseconds, and 66 milliseconds, etc.).
[0027] The adjustment unit 13 adjusts the setting information based on future bandwidth information and video information, and compresses the video information based on the setting information. The setting information is information indicating at least one of the resolution of the target image (1 frame) included in the video and the frame rate of the target image. Resolution is an indicator that indicates the resolution of the target image, for example, an indicator that indicates the field of view of the target image. For example, resolutions include HD, FHD, and 4K. The higher the resolution, the more detailed the target image becomes and the larger the bitrate (amount of data) of the target image becomes. Frame rate is an indicator that indicates the number of frames displayed per second. The higher the frame rate, the smoother the video becomes and the larger the bitrate of the video becomes. The frame rate of the target image is the number of images displayed per second in a video containing the target image, for example, it corresponds to the number of frames when the frame interval between the timing when the target image is displayed and the timing when the immediately preceding consecutive image is displayed lasts for one second. For example, the frame rate of the target image is set by setting the time interval between the target image and the immediately preceding consecutive image.
[0028] More specifically, the adjustment unit 13 adjusts the setting information so that the available bandwidth of the future network indicated by the future bandwidth information does not exceed the data amount of the target image indicated by the video information. For example, if the available bandwidth of the future network indicated by the future bandwidth information is 5 Mbps, the adjustment unit 13 adjusts at least one of the resolution and frame rate of the target image so that the data amount of the target image does not exceed 5 Mbps. The adjustment unit 13 compresses (encodes) the target image using a lossless encoding scheme according to at least one of the adjusted resolution and frame rate.
[0029] The adjustment unit 13 adjusts the setting information based on the amount of data (average amount of data) for each resolution of the target image (1 frame) indicated by the future bandwidth information and the video information. In this case, the adjustment unit 13 changes the resolution of the target image to be encoded next based on the amount of data for each resolution of the target image, so that the amount of data of the target image to be encoded next does not exceed the available bandwidth of the future network indicated by the future bandwidth information.
[0030] The adjustment unit 13 adjusts the setting information based on future bandwidth information and camera 101 operation information. In this case, when the adjustment unit 13 determines that there is no operation of the camera 101 based on the camera 101 operation information, it reduces the frame rate of the next target image (1 frame) so that the amount of data of the next target image (1 frame) to be encoded does not exceed the available bandwidth of the future network indicated by the future bandwidth information. As a result, when there is no operation of the camera 101, the video captured by the camera 101 becomes almost a still image, so even if the frame rate is reduced and compressed, the impact on the decoding process of the video including the target image is small. Therefore, the adjustment unit 13 can lower the encoding bitrate while maintaining the resolution of the video and suppressing the impact on the decoding process of the video.
[0031] The adjustment unit 13 adjusts the setting information based on the future bandwidth information and video information each time the estimation unit 12 estimates the future bandwidth information. The predetermined interval is a preset time interval. As an example, the adjustment unit 13 adjusts the setting information at time intervals smaller than several hundred milliseconds (e.g., 300 milliseconds). The adjustment unit 13 adjusts the setting information every few tens of milliseconds (e.g., 20 milliseconds, 33 milliseconds, and 66 milliseconds, etc.).
[0032] The transmitting unit 14 transmits video information related to the video whose setting information has been adjusted by the adjustment unit 13 via the network. More specifically, the transmitting unit 14 transmits the video information to the relay device 20 via the network. The transmitting unit 14 also transmits the data packet to the relay device 20 so that the acquisition unit 11 can obtain an acknowledgment of the data packet and acquire status information based on the acknowledgment.
[0033] The imaging unit 15 is a functional unit for generating video information. The imaging unit 15 captures video using various known methods and generates video information. The control unit 16 controls the imaging unit 15.
[0034] Next, the relay device 20 is configured to include a communication unit 21 and a decoding unit 22 as functional components. The communication unit 21 corresponds to the transmitting / receiving terminal 201 described above. The decoding unit 22 corresponds to the decoder 202 described above. The communication unit 21 receives video information from the transmitting device 10 via the network. The communication unit 21 also transmits distribution information (acknowledgment of data packets) to the transmitting device 10 so that the acquisition unit 11 of the transmitting device 10 can acquire status information.
[0035] The decoding unit 22 decodes the video information output from the communication unit 21. The decoding unit 22 transmits the decoded video information to the control device 30. More specifically, the decoding unit 22 converts the video information output from the communication unit 21 into video information corresponding to the setting information before it is adjusted by the adjustment unit 13 of the transmission device 10. That is, the decoding unit 22 decodes (decodes) the video information using a lossless encoding scheme and restores it to the video information before it was compressed by the adjustment unit 13, based on the aspect ratio and frame rate information of the frames contained in the video information. The method for decoding the video information can be implemented by various known methods.
[0036] Figure 4 is a flowchart showing an example of the transmission method processing performed by the transmitting device 10. Referring to Figure 4, an example of the transmission method processing performed by the transmitting device 10 will be explained.
[0037] First, the transmitting device 10 acquires initial configuration information (for example, frame resolution and frame rate) (step S101). Next, the transmitting device 10 sends and receives distribution information (step S102). Subsequently, the transmitting device 10 acquires first status information based on the distribution information (step S103). In the process of step S102, the transmitting device 10 performs the transmission of the data packets and the reception of acknowledgments for the data packets as described above. In the process of step S103, the transmitting device 10 acquires first status information based on acknowledgments for the data packets.
[0038] Next, the transmitting device 10 acquires video information (step S104). Next, the transmitting device 10 sends and receives new distribution information (step S105). Next, the transmitting device 10 converts the first state information acquired immediately before into second state information, and acquires new first state information based on the new distribution information (step S106). Next, the transmitting device 10 acquires past bandwidth information (step S107). In this way, in the acquisition steps corresponding to steps S104 to S107, the transmitting device 10 acquires video information, state information, and past bandwidth information. To explain using the above-mentioned equation (1), the state information corresponds to RTT(t-1) and RTT(t), and the past bandwidth information corresponds to transmission TP(t-1).
[0039] Next, the transmitting device 10 estimates future bandwidth information based on state information including first state information and second state information, and past bandwidth information (step S108: estimation step). Subsequently, the transmitting device 10 adjusts the setting information based on the bandwidth information, and the compressed video information based on the setting information is transmitted to the relay device 20 (step S109: adjustment step and transmission step). For example, in the process of step S108, the transmitting TP(t+1) is calculated based on RTT(t-1) and RTT(t) and transmitting TP(t-1) using the above-described equation (1).
[0040] Next, the transmitting device 10 determines whether or not the transmission of all videos has been completed (step S110). If the transmitting device 10 determines that the transmission of all videos has not been completed (step S110: NO), the process proceeds to step S104. If the transmitting device 10 determines that the transmission of all videos has been completed (step S110: YES), the transmission process ends.
[0041] Next, the effects of a transmission device 10 and transmission method relating to one aspect of this disclosure will be described. The transmission device 10 and transmission method estimate future bandwidth information relating to future network bandwidth based on status information indicating the network state and past bandwidth information relating to past network bandwidth. Based on the future bandwidth information, setting information indicating at least one of the resolution of the target image and the frame rate of the target image is adjusted, and video information is compressed based on the setting information. With this configuration, even if the network bandwidth changes, the changed bandwidth can be estimated with high accuracy. As a result, by compressing video information according to setting information based on future bandwidth information indicating the changed bandwidth, it becomes possible to smoothly play the video at the receiving device of the video information.
[0042] The above effects will be explained in detail. Conventionally, in order to broadcast video from a live broadcast site, it was necessary to compress the video information and transmit it to the broadcasting station using a dedicated equipment capable of transmitting and receiving radio signals transmitted and received by a broadcasting radio station (FPU: Field Pick-up Unit) and broadcasting radio stations. In this case, significant time, human, and financial costs were required. In contrast, it is conceivable to compress the video information into data with a data volume suitable for mobile communication and transmit the video information to the broadcasting station via a public mobile network. However, in this case, if the available bandwidth of the public mobile network etc. changes, conventional encoders (e.g., ST2110) alone have difficulty responding to such changes in available bandwidth, making it difficult to transmit video smoothly and with low latency. Consequently, it was difficult to play the video smoothly at the receiving terminal (for example, video stuttering would occur). In contrast, the transmission device 10 according to one aspect of this disclosure can accurately estimate the available bandwidth in real time, taking into account changes in the available bandwidth of the public mobile network etc. As a result, it is possible to control the available bandwidth of the public mobile network etc. in real time so that the video bitrate does not exceed it. Furthermore, since broadcasting radio stations and dedicated equipment capable of transmitting and receiving radio signals transmitted and received by broadcasting radio stations are not required, the time, personnel, and financial costs for transmitting video information to broadcasting stations can be reduced. In other words, video information can be transmitted from the relay site to the broadcasting station with less time, personnel, and financial costs (with less preparation and lower costs).
[0043] The adjustment unit 13 adjusts the setting information so that the available bandwidth of the future network indicated by the future bandwidth information does not exceed the amount of data of the target image indicated by the video information. In this case, it becomes more reliable that the video can be played back smoothly at the receiving device.
[0044] The state information includes first state information related to past network conditions, and second state information related to network conditions from a time further in the past than the past. In this case, the estimation unit 12 estimates the future bandwidth information further based on the result of comparing the first state information and the second state information, so that when the available bandwidth of the network changes, the available bandwidth after the change can be estimated with higher accuracy.
[0045] The state information is information indicating a round-trip delay time from the transmission device 10, which is the device that is the transmission source of the moving image information, to the relay device 20, which is the device that is the transmission destination of the moving image information. In this case, the future bandwidth information can be estimated with higher accuracy.
[0046] The past bandwidth information includes information indicating the data amount of past images. In this case, the estimation unit 12 can generate the future bandwidth information with higher accuracy in consideration of the available bandwidth of the network when the past images were transmitted.
[0047] The estimation unit 12 estimates the future bandwidth information every several tens of milliseconds, and the adjustment unit 13 adjusts the setting information based on the future bandwidth information each time the estimation unit 12 estimates the future bandwidth information. In this case, the future bandwidth information is estimated at intervals shorter than several hundred milliseconds, the setting information is adjusted based on the future bandwidth information, and the moving image is compressed based on the setting information. Thereby, the moving image information can be transmitted to the outside of the transmission device 10 while more reliably maintaining a state where the data amount of the target image is smaller than the available bandwidth of the network. As a result, it becomes possible more reliably to cause the device at the transmission destination to smoothly reproduce the moving image.
[0048] In the above embodiment, after adjusting the setting information, the adjustment unit 13 may adjust a correction value for correcting the setting information when adjusting the setting information next time. In this case, as shown in FIG. 5, after adjusting the setting information in the process of step S109, the correction value α in the above-described formula (1) may be adjusted. For example, the adjustment unit 13 estimates the estimated transmission TP(t+1) based on the actual RTT(t-1), transmission TP(t-1), and RTT(t). The adjustment unit 13 uses both the difference between the transmission TP(t-1) and the RTT(t-1) and the difference between the transmission TP(t-1) and the RTT(t) to estimate the estimated RTT(t+1) based on past trends. The adjustment unit 13 calculates a more appropriate calculated transmission TP(t+1) based on the deviation between the estimated RTT(t+1) and the actual RTT(t+1). The adjustment unit 13 adjusts the value of α when determining the data amount of the image (frame) at t+2, based on the difference between the estimated transmission TP(t+1) and the calculated transmission TP(t+1). Such adjustment has the effect of reducing the estimation deviation.
[0049] In the above embodiment, the acquisition unit 11 may acquire operation information for operating the imaging unit 15, and the control unit 16 may control the imaging unit 15 based on the operation information. In this case, in the example shown in FIG. 1, the control device 30 may include a transceiving terminal 301, a display terminal 302, and an operation terminal 303. The transceiving terminal 301 is, for example, a terminal or router capable of 5G communication. The transceiving terminal 301 causes the display terminal 302 to display the video signal received from the relay device 20.
[0050] In the example shown in FIG. 2, the control device 30 may include a communication unit 31, a display unit 32, and an operation reception unit 33. The communication unit 31 acquires the information received by the transceiving terminal 301. The display unit 32 displays the information acquired by the communication unit 31 on the display terminal 302. The operation reception unit 33 receives an operation input by a user who operates the control device 30 (for example, a staff member of a broadcasting station) using the operation terminal 303, and generates operation information. The communication unit 31 transmits the operation information output from the operation reception unit 33 to the transmission device 10.
[0051] The acquisition unit 11 may acquire operation information. The control unit 16 may control the imaging unit 15 based on the operation information acquired by the acquisition unit 11. This makes it possible for a person located away from the transmitting device 10 to operate the imaging unit 15 of the transmitting device 10. For example, a user operating the control device 30 can remotely control the imaging unit 15.
[0052] In the above embodiment, the transmitting device 10 corresponds to the transmission system and has an acquisition unit 11, an estimation unit 12, an adjustment unit 13, and a transmitting unit 14, but is not limited to this. For example, the transmitting device 10 and the relay device 20 may correspond to the transmission system. In this case, the transmitting device 10 may have an acquisition unit 11, an adjustment unit 13, and a transmitting unit 14, and the relay device 20 may have an estimation unit 12.
[0053] For example, the information processing system 1 may be a system used by a business that provides videos to users using multiple terminals. The transmission device 10 may be a device used by a business or individual that provides videos (for example, a distributor that delivers content to viewers using a live streaming platform or social media). The relay device 20 may be a server owned by a company that distributes videos on behalf of others (for example, a cloud server). The multiple terminals are terminals owned by each of the multiple users, and may be, for example, 5G-enabled terminals and personal computers (PCs).
[0054] In such cases, the encoding in the adjustment unit 13 can be controlled by setting information from the relay device 20 (cloud side) that indicates at least one of the resolution and frame rate of the target image. Furthermore, in low-latency video transmission, locating the available bandwidth estimation function and the video transmission function on the relay device 20 (on the cloud) facilitates the modernization of low-latency transmission technology. Moreover, when the relay device 20 provides video editing functionality to the transmission device 10 via a web application or the like, video editing suitable for low-latency transmission becomes easier, and it becomes possible to edit the low-latency transmitted video to make it easier to view before displaying it on the receiving terminal, without requiring any special preparation on the receiving terminal side.
[0055] Alternatively, for example, only the relay device 20 may be compatible with the transmission system. In this case, the relay device 20 may have an acquisition unit 11, an estimation unit 12, an adjustment unit 13, and a transmission unit 14. In this case, the transmission unit 14 of the relay device 20 may transmit video information to multiple terminals as described above, rather than to the control device 30. The network may be a network between the relay device 20 and each of the multiple terminals.
[0056] Furthermore, for example, the relay device 20 and the multiple terminals may correspond to the transmission system. The relay device 20 may have an acquisition unit 11, an adjustment unit 13, and a transmission unit 14, and each of the multiple terminals may have an estimation unit 12. In this case as well, the transmission unit 14 of the relay device 20 may transmit video information to the multiple terminals as described above, rather than to the control device 30.
[0057] The transmission system and transmission method of the present disclosure have the following configurations: [1] A transmission system for transmitting video over a network, comprising: an acquisition unit that acquires video information indicating a target image which is an image to be compressed among a plurality of images included in the video, state information relating to the state of the network, and past bandwidth information relating to the past available bandwidth of the network; an estimation unit that estimates future bandwidth information relating to the future available bandwidth of the network based on the state information and the past bandwidth information; an adjustment unit that adjusts setting information indicating at least one of the resolution of the target image and the frame rate of the target image based on the future bandwidth information and the video information, and compresses the video information based on the setting information; and a transmission unit that transmits the video information over the network. [2] The transmission system according to [1], wherein the adjustment unit adjusts the setting information so that the amount of data of the target image indicated by the video information does not exceed the future available bandwidth of the network indicated by the future bandwidth information. [3] The transmission system according to [1] or [2], wherein the state information includes first state information relating to the past state of the network and second state information relating to the state of the network even further in the past than the past. [4] The transmission system according to any one of [1] to [3], wherein the status information is information indicating the round-trip delay time from the device that transmits the video information to the device that receives the video information. [5] The transmission system according to any one of [1] to [4], wherein the past bandwidth information includes information indicating the amount of data of past images which are videos that the transmission unit has transmitted in the past. [6] The transmission system according to any one of [1] to [5], further comprising: an imaging unit for generating the video information; and a control unit for controlling the imaging unit, wherein the acquisition unit acquires operation information for operating the imaging unit; and the control unit controls the imaging unit based on the operation information. [7] The transmission system according to any one of [1] to [6], wherein the estimation unit estimates the future bandwidth information every tens of milliseconds; and the adjustment unit adjusts the setting information based on the future bandwidth information and the video information each time the estimation unit estimates the future bandwidth information.[8] A transmission method performed by a transmission system, comprising: an acquisition step in which the transmission system acquires video information indicating a target image which is an image to be compressed from among a plurality of images included in a video, state information relating to the state of the network, and past bandwidth information relating to the past available bandwidth of the network; an estimation step in which the transmission system estimates future bandwidth information relating to the future available bandwidth of the network based on the state information and the past bandwidth information; an adjustment step in which the transmission system adjusts setting information indicating at least one of the resolution of the target image and the frame rate of the target image based on the future bandwidth information and the video information, and compresses the video information based on the setting information; and a transmission step in which the transmission system transmits the video information over the network.
[0058] Next, the hardware configuration of the transmitting device 10 described above will be explained with reference to Figure 6. Physically, the transmitting device 10 may be configured as a computer device including a processor 1001, memory 1002, storage 1003, communication device 1004, input device 1005, output device 1006, bus 1007, etc.
[0059] In the following explanation, the term "device" can be replaced with "circuit," "device," "unit," etc. The hardware configuration of the transmitting device 10 may include one or more of the devices shown in the figure, or it may be configured to omit some of the devices.
[0060] Each function in the transmitting device 10 is realized by loading predetermined software (programs) onto hardware such as the processor 1001 and memory 1002, causing the processor 1001 to perform calculations and control communication by the communication device 1004 and the reading and / or writing of data to the memory 1002 and storage 1003.
[0061] The processor 1001 controls the entire computer, for example, by running the operating system. The processor 1001 may consist of a central processing unit (CPU) that includes interfaces with peripheral devices, control devices, arithmetic units, registers, etc. For example, control functions such as the estimation unit 12 of the transmission device 10 may be implemented by the processor 1001.
[0062] Furthermore, the processor 1001 reads programs (program code), software modules, and data from the storage 1003 and / or communication device 1004 into the memory 1002, and executes various processes accordingly. The program used is one that causes the computer to execute at least a part of the operations described in the above embodiment.
[0063] For example, control functions such as the estimation unit 12 of the transmitting device 10 may be stored in memory 1002 and implemented by a control program running on processor 1001, and other functional blocks may be implemented similarly. Although the above-described processes have been explained as being executed by one processor 1001, they may be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented on one or more chips. The program may also be transmitted from a network via a telecommunications line.
[0064] The memory 1002 is a computer-readable recording medium and may consist of at least one of the following: ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), RAM (Random Access Memory), etc. The memory 1002 may also be called a register, cache, main memory, etc. The memory 1002 can store executable programs (program code), software modules, etc., for carrying out the transmission method according to one embodiment of the present disclosure.
[0065] The storage 1003 is a computer-readable recording medium and may consist of at least one of the following: an optical disc such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital multipurpose disk, a Blu-ray® disc), a smart card, flash memory (e.g., a card, a stick, a key drive), a floppy® disk, a magnetic strip, etc. The storage 1003 may also be called an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, server, or other suitable medium including memory 1002 and / or storage 1003.
[0066] The communication device 1004 is hardware (transceiver / receiver device) for communicating between computers via a wired and / or wireless network, and is also referred to as a network device, network controller, network card, communication module, etc.
[0067] The input device 1005 is an input device that accepts input from an external source (e.g., a keyboard, mouse, microphone, switch, button, sensor, etc.). The output device 1006 is an output device that outputs to an external source (e.g., a display, speaker, LED lamp, etc.). The input device 1005 and the output device 1006 may be configured as an integrated unit (e.g., a touch panel).
[0068] Furthermore, each device, such as the processor 1001 and the memory 1002, is connected by a bus 1007 for communicating information. The bus 1007 may consist of a single bus, or different buses may be used for communication between devices.
[0069] Furthermore, the transmitting device 10 may be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array), and some or all of each functional block may be realized by such hardware. For example, the processor 1001 may be implemented using at least one of these hardware components.
[0070] Although this embodiment has been described in detail above, it will be clear to those skilled in the art that this embodiment is not limited to the embodiments described herein. This embodiment can be implemented as a modified and altered form without departing from the spirit and scope of this disclosure as defined by the claims. Therefore, the description herein is for illustrative purposes only and is not intended to be restrictive in any way to this embodiment.
[0071] Each aspect / embodiment described herein may be applied to systems utilizing LTE (Long Term Evolution), LTE-A (LTE-Advanced), SUPER 3G, IMT-Advanced, 4G, 5G, FRA (Future Radio Access), W-CDMA®, GSM®, CDMA2000, UMB (Ultra Mobile Broad-band), IEEE 802.11 (Wi-Fi), IEEE 802.16 (WiMAX), IEEE 802.20, UWB (Ultra-Wide Band), Bluetooth®, and other appropriate systems, and / or next-generation systems extended based thereon.
[0072] The processing procedures, sequences, flowcharts, etc., of each aspect / embodiment described herein may be reordered, provided they are consistent with each other. For example, the methods described herein present the elements of various steps in an exemplary order and are not limited to that specific order.
[0073] Input and output information may be stored in a specific location (e.g., memory) or managed in a management table. Input and output information may be overwritten, updated, or appended to. Output information may be deleted. Input information may be transmitted to other devices.
[0074] The determination may be made by a value represented by one bit (0 or 1), by a boolean value (true or false), or by a numerical comparison (for example, a comparison with a predetermined value).
[0075] Each aspect / embodiment described herein may be used individually, in combination, or switched between during implementation. Furthermore, notification of predetermined information (e.g., notification that "X is") is not limited to explicit notification, but may also be implicit (e.g., by not providing such notification).
[0076] Software should be broadly interpreted to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, and so on, whether they are called software, firmware, middleware, microcode, hardware description languages, or by any other name.
[0077] Furthermore, software, instructions, etc., may be transmitted and received via a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies such as coaxial cable, fiber optic cable, twisted pair, and digital subscriber lines (DSL) and / or wireless technologies such as infrared, radio, and microwave, these wired and / or wireless technologies are included in the definition of a transmission medium.
[0078] The information, signals, etc. described herein may be represented using any one of various different techniques. For example, the data, instructions, commands, information, signals, bits, symbols, chips, etc., which may be referred to throughout the above description, may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.
[0079] In addition, terms used herein and / or terms necessary for understanding this specification may be replaced with terms having the same or similar meanings.
[0080] Furthermore, the information, parameters, etc., described herein may be expressed as absolute values, relative values from a given value, or as corresponding other information.
[0081] Communication terminals may also be referred to by those skilled in the art as mobile communication terminals, subscriber stations, mobile units, subscriber units, wireless units, remote units, mobile devices, wireless devices, wireless communication devices, remote devices, mobile subscriber stations, access terminals, mobile terminals, wireless terminals, remote terminals, handsets, user agents, mobile clients, clients, or several other appropriate terms.
[0082] As used herein, the phrase "based on" does not mean "based solely on" unless otherwise specified. In other words, the phrase "based on" means both "based solely on" and "based at least on."
[0083] Where the terms “first,” “second,” etc., are used herein, no reference to those elements shall generally limit the quantity or order of those elements. These terms may be used herein as a convenient way to distinguish between two or more elements. Accordingly, references to the first and second elements shall not imply that only two elements may be employed therein, or that the first element must precede the second element in any way.
[0084] To the extent that “include,” “including,” and their variations are used herein or in the claims, these terms are intended to be inclusive, as is the term “comprising.” Furthermore, the term “or” as used herein or in the claims is not intended to be exclusive OR.
[0085] In this specification, unless the context or technically it is clear that only one device exists, the term also includes multiple devices.
[0086] Throughout this disclosure, unless the context clearly indicates a singular number, the terms shall include plural ones.
[0087] 10...Transmitting device (transmission system), 11...Acquisition unit, 12...Estimation unit, 13...Adjustment unit, 14...Transmitting unit, 15...Imaging unit, 16...Control unit, 1001...Processor, 1002...Memory, 1003...Storage, 1004...Communication device, 1005...Input device, 1006...Output device, 1007...Bus.
Claims
1. A transmission system for transmitting video over a network, comprising: an acquisition unit that acquires video information indicating a target image which is to be compressed from among a plurality of images contained in the video, state information relating to the state of the network, and past bandwidth information relating to the past available bandwidth of the network; an estimation unit that estimates future bandwidth information relating to the future available bandwidth of the network based on the state information and the past bandwidth information; an adjustment unit that adjusts setting information indicating at least one of the resolution of the target image and the frame rate of the target image based on the future bandwidth information and the video information, and compresses the video information based on the setting information; and a transmission unit that transmits the video information over the network.
2. The transmission system according to claim 1, wherein the adjustment unit adjusts the setting information so that the available bandwidth of the future network indicated by the future bandwidth information does not exceed the amount of data of the target image indicated by the video information.
3. The transmission system according to claim 1, wherein the status information includes first status information relating to a past state of the network and second status information relating to a state of the network even further in the past than the past.
4. The transmission system according to claim 1, wherein the status information is information indicating the round-trip delay time from the device that transmits the video information to the device that receives the video information.
5. The transmission system according to claim 1, wherein the past bandwidth information includes information indicating the amount of data of past images, which are videos, previously transmitted by the transmission unit.
6. The transmission system according to claim 1, further comprising: an imaging unit for generating the video information; and a control unit for controlling the imaging unit, wherein the acquisition unit acquires operation information for operating the imaging unit, and the control unit controls the imaging unit based on the operation information.
7. The transmission system according to claim 1, wherein the estimation unit estimates the future bandwidth information every tens of milliseconds, and the adjustment unit adjusts the setting information based on the future bandwidth information and the video information each time the estimation unit estimates the future bandwidth information.
8. A transmission method performed by a transmission system, comprising: an acquisition step in which the transmission system acquires video information indicating a target image which is an image to be compressed among a plurality of images included in a video, state information relating to the state of the network, and past bandwidth information relating to the past available bandwidth of the network; an estimation step in which the transmission system estimates future bandwidth information relating to the future available bandwidth of the network based on the state information and the past bandwidth information; an adjustment step in which the transmission system adjusts setting information indicating at least one of the resolution of the target image and the frame rate of the target image based on the future bandwidth information and the video information, and compresses the video information based on the setting information; and a transmission step in which the transmission system transmits the video information over the network.