Server device and distribution method

Abstract

This server device comprises: a video generation unit that generates a multiangle video by synthesizing, into one video, videos of a plurality of angles obtained by capturing an imaging target from different directions; a voice comment acquisition unit that acquires, from a plurality of terminal devices, voice comment information indicating voice comments made by users of the plurality of terminal devices when viewing the multiangle video; a voice comment generation unit that generates, on the basis of the voice comment information acquired by the voice comment acquisition unit, superimposition information to be superimposed on the multiangle video; and a distribution unit that distributes the multiangle video, the voice comment information, and the superimposition information to the plurality of terminal devices.

Description

Server Apparatus and Distribution Method

[0001] The present disclosure relates to a server apparatus and a distribution method.

[0002] Video distribution services that distribute moving images are known. For example, Patent Document 1 discloses a technology that allows both a performer and a user to experience a sense of presence by bidirectionally communicating information such as audio in real time between the performer and the user.

[0003] Japanese Unexamined Patent Publication No. 2022-46878

[0004] Meanwhile, in a service that distributes multi-angle video including video from a plurality of angles, a user can select and view video of a desired angle from among the plurality of angle videos. Normally, since information such as priority is not added to the plurality of videos, the user may be unsure which angle of video to select from among the plurality of angle videos. It is desired to provide a user with an index for selecting a video.

[0005] An object of the present disclosure is to provide a server apparatus and a distribution method capable of providing an index for a user to select a video to view from among a plurality of videos of different angles.

[0006] The server apparatus of the present disclosure comprises: a video generation unit that synthesizes a plurality of videos of different angles, obtained by photographing a photographing target from different directions, into one video to generate a multi-angle video; a voice comment acquisition unit that acquires voice comment information indicating voice comments uttered by users of terminal devices when the users view the multi-angle video, from a plurality of terminal devices; a voice comment generation unit that generates superimposition information to be superimposed on the multi-angle video based on the voice comment information acquired by the voice comment acquisition unit; and a distribution unit that distributes the multi-angle video, the voice comment information, and the superimposition information to the plurality of terminal devices.

[0007] The distribution method of this disclosure includes the steps of: generating a multi-angle video by combining multiple angle videos taken of a subject from different directions into a single video; acquiring audio comment information from multiple terminal devices indicating audio comments made by users of the terminal devices when they viewed the multi-angle video; generating superimposed information to be superimposed on the multi-angle video based on the acquired audio comment information; and distributing the multi-angle video, the audio comment information, or the superimposed information to the multiple terminal devices.

[0008] This disclosure provides indicators for users to select which video to watch from among multiple camera angles.

[0009] Figure 1 is a diagram showing an example configuration of a distribution system according to each embodiment. Figure 2 is a block diagram showing an example configuration of a server device according to each embodiment. Figure 3 is a block diagram showing an example configuration of a terminal device according to each embodiment. Figure 4 is a diagram illustrating a first playback method of multi-angle video according to each embodiment. Figure 5 is a diagram illustrating a second playback method of multi-angle video according to each embodiment. Figure 6 is a diagram illustrating a method for outputting audio comments according to each embodiment. Figure 7 is a flowchart showing the processing flow of the server device according to the first embodiment. Figure 8 is a flowchart showing the processing flow of the terminal device according to the first embodiment. Figure 9 is a flowchart showing the processing flow of the server device according to the second embodiment. Figure 10 is a flowchart showing the processing flow of the terminal device according to the second embodiment. Figure 11 is a flowchart showing the processing flow of the server device according to the third embodiment. Figure 12 is a flowchart showing the processing flow of the terminal device according to the third embodiment. Figure 13 is a flowchart showing the processing flow of the server device according to the fourth embodiment. Figure 14 is a flowchart showing the processing flow of the terminal device according to the fourth embodiment.

[0010] Embodiments relating to this disclosure will be described in detail below with reference to the attached drawings. However, this embodiment does not limit the disclosure, and in the following embodiments, the same parts are denoted by the same reference numerals to avoid redundant explanations.

[0011] [Embodiments] (Distribution System) Using Figure 1, examples of the configuration of the distribution system according to each embodiment will be described. Figure 1 is a diagram showing examples of the configuration of the distribution system according to each embodiment.

[0012] As shown in Figure 1, the distribution system 1 includes cameras 10-1 through 10-n (where n is an integer of 2 or more), a server device 12, and terminal devices 14-1 and 14-m (where m is an integer of 2 or more). When it is not necessary to distinguish between cameras 10-1 through 10-n, they are collectively referred to as camera 10. When it is not necessary to distinguish between terminal devices 14-1 through 14-m, they are collectively referred to as terminal device 14. However, the distribution system 1 may have only one terminal device 14. Camera 10, the server device 12, and the terminal device 14 are connected to each other via a network N so as to be able to communicate. The distribution system 1 transmits multi-angle video, which includes images of the target being filmed from multiple angles, to the terminal device 14. Here, the target being filmed by the distribution system 1 is not limited to one. For example, the distribution system 1 may transmit multi-angle video, which includes images of different targets being filmed at multiple locations in the same event, such as a gymnastics competition, using multiple cameras, to the terminal device 14.

[0013] Cameras 10-1 through 10-n are each installed to photograph the target object from different directions. Cameras 10-1 through 10-n are each installed in predetermined positions, for example. Cameras 10-1 through 10-n transmit the captured video to the server device 12 via the network N. The server device 12 generates a multi-angle video including the video received from cameras 10-1 through 10-n. The server device 12 transmits the generated multi-angle video to the terminal device 14. The terminal device 14 plays back the multi-angle video based on user input. In this disclosure, cameras 10-1 through 10-n capture a fixed field of view at their installation locations.

[0014] In this disclosure, the terminal device 14 transmits audio comments made by a user viewing multi-angle video to the server device 12. Based on the audio comments received from the terminal device 14, the server device 12 generates an index for the user to select the video angle to view from among the multiple angles included in the multi-angle video, and transmits it to the terminal device 14. This makes it easier for the user to select the video to view from among the multiple angles.

[0015] (Server Device) Using Figure 2, we will describe the configuration examples of the server device according to each embodiment. Figure 2 is a block diagram showing the configuration examples of the server device according to each embodiment.

[0016] As shown in Figure 2, the server device 12 comprises a communication unit 20, a storage unit 22, and a control unit 24. The server device 12 can be implemented, for example, by an information processing device such as a computer installed by a business operator that distributes multi-angle video.

[0017] The communication unit 20 is a communication interface that performs communication between the server device 12 and external devices. For example, the communication unit 20 performs communication between the server device 12 and the camera 10. For example, the communication unit 20 performs communication between the server device 12 and the terminal device 14.

[0018] The memory unit 22 stores various types of information. The memory unit 22 stores the calculation contents of the control unit 24 and information such as programs. The memory unit 22 includes at least one of the following: a main memory device such as RAM (Random Access Memory) and ROM (Read Only Memory), or an external memory device such as an HDD (Hard Disk Drive).

[0019] The control unit 24 controls each part of the server device 12. The control unit 24 includes, for example, an information processing device such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), and a storage device such as RAM or ROM. The control unit 24 executes a program that controls the operation of the server device 12 according to the present invention. The control unit 24 may be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 24 may be implemented by a combination of hardware and software.

[0020] The control unit 24 includes a video acquisition unit 30, an audio comment acquisition unit 32, a video generation unit 34, an audio comment generation unit 36, and a distribution unit 38.

[0021] The video acquisition unit 30 acquires video from each camera 10 via the communication unit 20. That is, the video acquisition unit 30 acquires video from multiple angles, capturing the subject from multiple different angles, via the communication unit 20. Here, the video acquired by the video acquisition unit 30 from each camera 10 also includes audio picked up by a microphone (not shown) provided in each camera 10. In other words, the video acquired by the video acquisition unit 30 also includes audio from multiple angles, capturing the subject from multiple different angles.

[0022] The video acquisition unit 30 acquires identification information from each camera 10 when acquiring video from each camera 10. The video acquisition unit 30 also acquires location information indicating the location where each camera 10 is installed when acquiring video from each camera 10.

[0023] The voice comment acquisition unit 32 acquires, via the communication unit 20, voice comments uttered by a user viewing multi-angle video from the terminal device 14, and angle identification information of the angle the user was viewing at the time the voice comments were uttered. The angle identification information is, for example, the identification information of the camera that filmed the video the user is viewing. The video identification information may also include information indicating the playback method of the video the user is viewing, as described later.

[0024] The video generation unit 34 synthesizes the multiple angle videos acquired by the video acquisition unit 30 into a single video to generate a multi-angle video. There is no particular limit to the number of videos included in the multi-angle video. Furthermore, the video generation unit 34 also includes determining the audio for an arbitrary angle to be played back from the audio of multiple angles acquired by the video acquisition unit 30.

[0025] The audio comment generation unit 36 ​​generates superimposed information to be superimposed on the multi-angle video based on the audio comments acquired by the audio comment acquisition unit 32. For example, the audio comment generation unit 36 ​​generates audio information from the audio comments acquired by the audio comment acquisition unit 32 and superimposes it on the multi-angle video as a secondary audio track.

[0026] The audio comment generation unit 36 ​​generates, for example, comment count information indicating the number of audio comments acquired by the audio comment acquisition unit 32 for each of the multiple angles of video included in the multi-angle video, and comment volume information indicating the volume level of the audio comments, as superimposed information.

[0027] The distribution unit 38 distributes the multi-angle video generated by the video generation unit 34, the audio comment acquisition unit 32, and the superimposed information generated by the audio comment generation unit 36 ​​to the terminal device 14 via the communication unit 20. If the superimposed information generated by the audio comment generation unit 36 ​​includes audio information generated from the audio comment acquisition unit 32, the distribution unit 38 does not need to distribute the audio comment to the terminal device 14.

[0028] (Terminal Device) Using Figure 3, we will describe the configuration examples of the terminal device according to each embodiment. Figure 3 is a block diagram showing the configuration examples of the terminal device according to each embodiment.

[0029] As shown in Figure 3, the terminal device 14 includes an input unit 40, a microphone 42, a display unit 44, an audio output unit 46, a communication unit 48, a storage unit 50, and a control unit 52. The terminal device 14 can be implemented using a network terminal capable of playing video and audio, such as a computer, tablet, smartphone, or television owned by the user.

[0030] The input unit 40 receives various input operations for the terminal device 14. The input unit 40 outputs an operation signal to the control unit 52 corresponding to the received input operation. The input unit 40 includes, for example, a touch panel, buttons, switches, etc. When a touch panel is used as the input unit 40, the input unit 40 is arranged on the display unit 44.

[0031] Microphone 42 detects the sound emitted by the user of the terminal device 14. Microphone 42 detects the sound emitted by the user while they are viewing multi-angle video streamed from the server device 12 on the terminal device 14. Microphone 42 converts the detected sound into an audio signal.

[0032] The display unit 44 displays various types of video. For example, the display unit 44 displays multi-angle video distributed from the server device 12. For example, the display unit 44 displays information about multiple user voice comments superimposed on the multi-angle video. The display unit 44 is a display including, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display.

[0033] The audio output unit 46 is a speaker that outputs various types of audio. For example, the audio output unit 46 outputs audio from any angle determined to be playing multi-angle video. For example, the audio output unit 46 outputs audio comments from multiple users superimposed on the multi-angle video.

[0034] The communication unit 48 is a communication interface that performs communication with the server device 12.

[0035] The memory unit 50 stores various types of information. The memory unit 50 stores the calculation contents of the control unit 52 and information such as programs. The memory unit 50 includes, for example, RAM, a main memory such as ROM, and an external memory such as an HDD, at least one of these.

[0036] The control unit 52 controls each part of the terminal device 14. The control unit 52 includes, for example, an information processing device such as a CPU or MPU, and a storage device such as RAM or ROM. The control unit 52 executes a program that controls the operation of the terminal device 14 according to the present invention. The control unit 52 may be implemented by an integrated circuit such as an ASIC or FPGA. The control unit 52 may be implemented by a combination of hardware and software.

[0037] The control unit 52 comprises an acquisition unit 60, a playback unit 62, a selection unit 64, an audio acquisition unit 66, and a communication control unit 68.

[0038] The acquisition unit 60 acquires multi-angle video from the server device 12 via the communication unit 48. The acquisition unit 60 also acquires information regarding audio comments superimposed on the multi-angle video from the server device 12 via the communication unit 48.

[0039] The playback unit 62 plays back the multi-angle video acquired by the acquisition unit 60 and displays it on the display unit 44. The playback unit 62 displays information regarding audio comments for each of the multiple angles of video included in the multi-angle video acquired by the acquisition unit 60 on the display unit 44.

[0040] The playback unit 62 outputs the audio of the multi-angle video acquired by the acquisition unit 60 from the audio output unit 46. The playback unit 62 also outputs the audio commentary superimposed on the multi-angle video acquired by the acquisition unit 60 from the audio output unit 46.

[0041] The selection unit 64 selects, as angle identification information, the video to be viewed by the user, or the video playback method described later, from among the multiple videos included in the multi-angle video acquired by the acquisition unit 60. Specifically, the selection unit 64 selects a video based on the information input to the input unit 40.

[0042] The audio acquisition unit 66 controls the microphone 42 to detect audio. The audio acquisition unit 66 acquires the audio signal output from the microphone 42 as an audio comment.

[0043] The communication control unit 68 controls the communication unit 48 to execute communication. For example, the communication control unit 68 controls the communication unit 48 to transmit the audio comment detected by the microphone 42 and the angle identification information of the video selected by the selection unit 64 to the server device 12.

[0044] (First Playback Method for Multi-angle Video) A first playback method for multi-angle video according to each embodiment will be described with reference to Fig. 4. Fig. 4 is a diagram for explaining the first playback method for multi-angle video according to each embodiment.

[0045] As shown in Fig. 4, the multi-angle video 100 includes a first area 110, a second area 120, a third area 130, a fourth area 140, and a fifth area 150. The multi-angle video 100 is displayed on the display unit 44 of the terminal device 14.

[0046] The first area 110 is, for example, an area where video of a first angle captured by the camera 10-1 is displayed. The second area 120 is, for example, an area where video of a second angle captured by the camera 10-2 is displayed. The third area 130 is, for example, an area where video of a third angle captured by the camera 10-3 is displayed. The fourth area 140 is, for example, an area where video of a fourth angle captured by the camera 10-4 is displayed. The fifth area 150 is an area where the selected video from among the first-angle video to the fourth-angle video is displayed. The areas from the first area 110 to the fourth area 140 have small areas, while the fifth area 150 has a large area. In the example shown in Fig. 4, the video of the fourth angle is displayed in the fifth area 150. That is, in the first playback method, the selected video is displayed in an enlarged size.

[0047] In the first area 110, first comment information 111 is displayed. The first comment information 111 includes information about whether there is an audio comment for the video of the first angle. The first comment information 111 includes, for example, information about the number of audio comments acquired from the terminal device 14 of a user who is viewing the video of the first angle, and the volume of the audio comments. In the example shown in FIG. 4, the first comment information 111 is simply indicated as "0", but in practice, specific information regarding the number of audio comments for the video of the first angle and the volume of the audio comments may be displayed.

[0048] In the second area 120, second comment information 121 is displayed. The second comment information 121 includes information about whether there is an audio comment for the video of the second angle. The second comment information 121 includes, for example, information about the number of audio comments acquired from the terminal device 14 of a user who is viewing the video of the second angle, and the volume of the audio comments. In the example shown in FIG. 4, the second comment information 121 is simply indicated as "2", but in practice, specific information regarding the number of audio comments for the video of the second angle and the volume of the audio comments may be displayed.

[0049] In the third area 130, third comment information 131 is displayed. The third comment information 131 includes information about whether there is an audio comment for the video of the third angle. The third comment information 131 includes, for example, information about the number of audio comments acquired from the terminal device 14 of a user who is viewing the video of the third angle, and the volume of the audio comments. In the example shown in FIG. 4, the third comment information 131 is simply indicated as "0", but in practice, specific information regarding the number of audio comments for the video of the third angle and the volume of the audio comments may be displayed.

[0050] The fourth area 140 displays the fourth comment information 141. The fourth comment information 141 includes information regarding the presence or absence of audio comments for the video from the fourth angle. For example, the fourth comment information 141 includes information regarding the number of audio comments obtained from the terminal device 14 of the user viewing the video from the fourth angle, and the volume of the audio comments. In the example shown in Figure 4, the fourth comment information 141 is simply shown as "1," but in reality, specific information regarding the number of audio comments for the video from the fourth angle and the volume of the audio comments may be shown.

[0051] The fifth comment information 151 includes the number of audio comments obtained from the terminal device 14 of a user who is viewing multi-angle video using the second playback method described later, and information regarding the volume of the audio comments.

[0052] The sixth comment information 161 includes information about the total number of audio comments for users viewing the multi-angle video from the first angle to the fourth angle, and for users viewing the multi-angle video using the second playback method. In the example shown in Figure 4, the sixth comment information 161 is simply shown as "4," but in reality, specific information regarding the total number of audio comments for the first to fourth angle videos may be shown.

[0053] Here, the first comment information 111 to the sixth comment information 161 may be information based on the playback time at which the audio comment was acquired. In other words, the audio comment acquisition unit 32 further acquires from the terminal device 14 the time at which the playback time corresponds to the audio comment. The terminal device 14 then counts and displays the audio comment as a number only if the playback time from the playback unit 62 corresponds to the time at which the audio comment was acquired.

[0054] The playback method switching key 171 is a key indicator for switching the playback method between the first playback method and the second playback method. When the playback method switching key 171 is pressed on the user's terminal device 14, the multi-angle video 100 is switched from the first playback method to the second playback method and displayed.

[0055] The playback unit 62 may output audio comments superimposed on the video of the angle displayed in the fifth region 150 from the audio output unit 46. The playback unit 62 may display the audio comments superimposed on the video of the angle displayed in the fifth region 150 in the fifth region 150. The playback unit 62 may also provide a separate region for displaying audio comments, distinct from the fifth region 150.

[0056] (Second Playback Method for Multi-Angle Video) The second playback method for multi-angle video according to each embodiment will be explained using Figure 5. Figure 5 is a diagram for explaining the second playback method for multi-angle video according to each embodiment.

[0057] As shown in Figure 5, the multi-angle video 200 includes a first region 210, a second region 220, a third region 230, and a fourth region 240.

[0058] The first region 210 is the region where, for example, the image of the first angle captured by camera 10-1 is displayed. The second region 220 is the region where, for example, the image of the second angle captured by camera 10-2 is displayed. The third region 230 is the region where, for example, the image of the third angle captured by camera 10-3 is displayed. The fourth region 240 is the region where, for example, the image of the fourth angle captured by camera 10-4 is displayed. The areas of the first region 210 to the fourth region 240 are the same.

[0059] The first area 210 displays the first comment information 211. The first comment information 211 includes information regarding the presence or absence of audio comments for the video at the first angle. For example, the first comment information 211 includes the number of audio comments obtained from the terminal device 14 of the user viewing the video at the first angle, and information regarding the volume of the audio comments. In the example shown in Figure 6, the first comment information 211 is simply shown as "0", but in reality, specific information regarding the number of audio comments for the video at the first angle and the volume of the audio comments may be shown.

[0060] The second area 220 displays the second comment information 221. The second comment information 221 includes information regarding the presence or absence of audio comments for the video from the second angle. For example, the second comment information 221 includes information regarding the number of audio comments obtained from the terminal device 14 of the user viewing the video from the second angle, and the volume of the audio comments. In the example shown in Figure 6, the second comment information 221 is simply shown as "2," but in reality, specific information regarding the number of audio comments for the video from the second angle and the volume of the audio comments may be shown.

[0061] Third comment information 231 is displayed in the third area 230. The third comment information 231 includes information regarding the presence or absence of audio comments for the video from the third angle. For example, the third comment information 231 includes information regarding the number of audio comments obtained from the terminal device 14 of the user viewing the video from the third angle, and the volume of the audio comments. In the example shown in Figure 6, the third comment information 231 is simply shown as "0", but in reality, specific information regarding the number of audio comments for the video from the third angle and the volume of the audio comments may be shown.

[0062] The fourth area 240 displays the fourth comment information 241. The fourth comment information 241 includes information regarding the presence or absence of audio comments for the video from the fourth angle. For example, the fourth comment information 241 includes information regarding the number of audio comments obtained from the terminal device 14 of the user viewing the video from the fourth angle, and the volume of the audio comments. In the example shown in Figure 6, the fourth comment information 241 is simply shown as "1", but in reality, specific information regarding the number of audio comments for the video from the fourth angle and the volume of the audio comments may be shown.

[0063] The fifth comment information 251 includes the number of audio comments obtained from the terminal device 14 of the user viewing the multi-angle video using the second playback method, and information regarding the volume of the audio comments.

[0064] The sixth comment information 261 includes information about the total number of audio comments for users viewing the multi-angle video from the first to the fourth video, and using the second playback method. In the example shown in Figure 6, the sixth comment information 261 is simply shown as "4," but in reality, it may show specific information about the total number of audio comments for the first to fourth angle videos.

[0065] The playback method switching key 271 is a key indicator for switching the playback method between the first playback method and the second playback method. When the playback method switching key 271 is pressed on the user's terminal device 14, the multi-angle video 100 is switched from the second playback method to the first playback method and displayed.

[0066] In the example shown in Figure 5, for example, the playback unit 62 may simultaneously output audio comments superimposed on the video from the first angle to the video from the fourth angle from the audio output unit 46.

[0067] (Voice Comment Output Method) The voice comment output method according to each embodiment will be explained using Figure 6. Figure 6 is a diagram illustrating the voice comment output method according to each embodiment.

[0068] For example, when the playback unit 62 plays audio commentary superimposed on video other than the video being viewed by the user among the video included in the multi-angle video, it may change the method of playing the audio commentary based on the positional relationship of each camera 10 that is shooting each video. The positional relationship of each camera 10 is a type of directional information.

[0069] For example, as shown in Figure 3, let's assume that the object 300 is being photographed using four cameras 10, from camera 10-1 to camera 10-4. Camera 10-1 is photographing the object 300 from the rear. Camera 10-2 is photographing the object 300 from the right. Camera 10-3 is photographing the object 300 from the front. Camera 10-4 is photographing the object 300 from the left.

[0070] In this case, the voice comment generation unit 36 ​​may generate superimposed audio information that changes the direction in which the voice comment is heard, based on the positional relationship from camera 10-1 to camera 10-4. For example, the voice comment generation unit 36 ​​may generate directional information indicating the direction from each camera 10 to the other camera 10 based on the positional relationship acquired from each camera 10, and generate superimposed audio information that changes the direction in which the voice comment is heard. In this case, the distribution unit 38 distributes audio information including directional information to the terminal device 14 via the communication unit 20, and does not need to distribute the voice comment acquired by the voice comment acquisition unit 32.

[0071] The playback unit 62 plays back audio comments based on audio information received from the server device 12, so that the direction from which the audio comments are heard changes. For example, suppose the user is viewing video from an angle where the subject 300 was filmed by camera 10-3. In this case, the playback unit 62 outputs audio from the audio output unit 46 so that the audio comments of a user viewing video from an angle where the subject 300 was filmed by camera 10-1 are heard from the front. The playback unit 62 also outputs audio from the audio output unit 46 so that the audio comments of a user viewing video from an angle where the subject 300 was filmed by camera 10-2 are heard from the left, and audio from the audio output unit 46 so that the audio comments of a user viewing video from an angle where the subject 300 was filmed by camera 10-4 are heard from the right. As a result, the direction of the audio comments matches the viewing direction in which the user is viewing the video, which enhances the sense of realism given to the user viewing the video.

[0072] Furthermore, the audio comment generation unit 36 ​​may generate superimposed audio information in which the direction from which the audio comment is heard changes based on the arrangement of the images from each angle included in the multi-angle video. For example, in the case of the multi-angle video 200, which is the second playback method shown in Figure 5, the audio comment generation unit 36 ​​may generate audio information in which the audio comment for the image displayed in the first region 210 is heard from the upper left, the audio comment for the image displayed in the second region 220 is heard from the upper right, the audio comment for the image displayed in the third region 230 is heard from the lower left, and the audio comment for the image displayed in the fourth region 240 is heard from the lower right.

[0073] In this case, the playback unit 62 outputs audio comments from the audio output unit 46 such that audio comments for the video displayed in the first area 210 are heard from the upper left, audio comments for the video displayed in the second area 220 are heard from the upper right, audio comments for the video displayed in the third area 230 are heard from the lower left, and audio comments for the video displayed in the fourth area 240 are heard from the lower right. This makes it easier for the user to understand the correspondence between each video and each audio comment, making it easier to select the video to watch. In addition, the presence of audio comments coming from different directions enhances the immersive experience for the user watching the video.

[0074] (Server Device Processing) The processing flow of the server device according to the first embodiment will be explained using Figure 7. Figure 7 is a flowchart showing the processing flow of the server device according to the first embodiment.

[0075] The video acquisition unit 30 acquires video from multiple angles (step S10). Specifically, the video acquisition unit 30 acquires video captured by each of the multiple cameras 10. Then, the process proceeds to step S12.

[0076] The video generation unit 34 generates a multi-angle video that includes video from multiple angles acquired by the video acquisition unit 30 (step S12). Then, the process proceeds to step S14.

[0077] The audio comment acquisition unit 32 acquires audio comment information from the terminal device 14 regarding audio comments on the video being viewed by the user (step S14). Then, the process proceeds to step S16.

[0078] The distribution unit 38 distributes the multi-angle video and audio commentary information to the terminal device 14 (step S16). Then, the process shown in Figure 7 is completed.

[0079] (Processing of the terminal device) The processing flow of the terminal device according to the first embodiment will be explained using Figure 8. Figure 8 is a flowchart showing the processing flow of the terminal device according to the first embodiment.

[0080] The acquisition unit 60 acquires multi-angle video and audio comment information from the server device 12 (step S20). Then, the process proceeds to step S22.

[0081] The playback unit 62 plays back the multi-angle video and audio comment information acquired by the acquisition unit 60 (step S22). Then, the process proceeds to step S24.

[0082] The selection unit 64 determines whether or not angle selection information has been acquired (step S24). Specifically, in the example shown in Figure 4, the selection unit 64 determines that angle selection information has been acquired when an operation to select a video to be viewed from among the videos displayed in the first region 110 to the fourth region 140 is input to the input unit 40 in the first playback method. The user can select the desired video, for example, using audio commentary information, which is played along with the multi-angle video, as an indicator. If it is determined that angle selection information has been acquired (step S24; Yes), the process proceeds to step S26. If it is not determined that angle selection information has been acquired (step S24; No), the process proceeds to step S28.

[0083] Step S24; If Yes, the playback unit 62 plays back the multi-angle video based on the angle selection information (Step S26). Specifically, in the example shown in Figure 4, the playback unit 62 displays the selected video from the videos displayed in the first region 110 to the fourth region 140 in the fifth region 150. Then, the process proceeds to step S30.

[0084] Step S24; If No, the regeneration unit 62 continues regeneration in the current state (step S28). Then, the process proceeds to step S30.

[0085] The audio acquisition unit 66 determines whether or not it has detected sound emitted by a user viewing the multi-angle video (step S30). Specifically, the audio acquisition unit 66 determines that sound has been detected if it has acquired an audio signal from the microphone 42. If it is determined that sound has been detected (step S30; Yes), the process proceeds to step S32. If it is determined that sound has not been detected (step S30; No), the process proceeds to step S34.

[0086] Step S30: If Yes, the audio signal acquired by the audio acquisition unit 66 is transmitted to the server device 12 via the communication control unit 68 as audio comment information together with the angle identification information (Step S32). Specifically, if the user is viewing the multi-angle video using the first playback method, the playback method and the identification information of the camera being viewed are transmitted to the server device 12 as angle identification information. If the user is viewing the multi-angle video using the second playback method, only the playback method is transmitted to the server device 12 as angle identification information. Then, the process proceeds to step S34.

[0087] The control unit 52 determines whether or not to terminate the process (step S34). Specifically, the control unit 52 determines to terminate the process when the user finishes watching the multi-angle video or when it receives an operation to turn off the power of the terminal device 14. If it is determined that the process should be terminated (step S34; Yes), the process shown in Figure 8 is terminated. If it is not determined that the process should be terminated (step S34; No), the process returns to step S20 and the above process is repeated.

[0088] In the first embodiment, the video included in the multi-angle video is played along with audio comments made by other users in response to the video. In the first embodiment, the number of comments and comment volume information are displayed for each angle of the video. This makes it easy for the user to select which video to view from among the videos included in the multi-angle video in the first embodiment.

[0089] [Second Embodiment] (Processing of Server Device) The processing flow of the server device according to the second embodiment will be explained using Figure 9. Figure 9 is a flowchart showing the processing flow of the server device according to the second embodiment.

[0090] The processes from step S40 to step S44 are the same as the processes from step S10 to step S14 shown in Figure 7, so their explanation will be omitted.

[0091] The voice comment generation unit 36 ​​generates superimposed information, which includes comment count information indicating the number of comments and comment volume information indicating the volume level of the comments, based on the voice comment acquisition information acquired by the voice comment acquisition unit 32 (step S46). Then, the process proceeds to step S48.

[0092] The distribution unit 38 distributes multi-angle video, audio comment information, comment count information, and comment volume information to the terminal device 14 (step S48). Then, the process shown in Figure 9 is completed.

[0093] (Processing of the terminal device) The processing flow of the terminal device according to the second embodiment will be explained using Figure 10. Figure 10 is a flowchart showing the processing flow of the terminal device according to the second embodiment.

[0094] The acquisition unit 60 acquires multi-angle video, audio comment information, comment count information, and comment volume information from the server device 12 (step S60). Then, the process proceeds to step S62.

[0095] The process in step S62 is the same as the process in step S22 shown in Figure 8, so its explanation will be omitted.

[0096] The playback unit 62 displays comment count information and comment volume information on the display unit 44 (step S64). Specifically, the playback unit 62 displays comment count information and comment volume information on the display unit 44 for each video included in the multi-angle video. For example, in the example shown in Figure 4, the playback unit 62 displays the first comment information 111, the second comment information 121, the third comment information 131, the fourth comment information 141, the fifth comment information 151, and the sixth comment information 161 on the display unit 44. This makes it easier for the user to select the video to watch based on the comment count information and comment volume information.

[0097] The selection unit 64 determines whether or not angle selection information has been acquired (step S66). Specifically, the selection unit 64 determines that angle selection information has been acquired when an operation to select an angle is input to the input unit 40. The selection unit 64 may select the video of the angle the user will view based on the number of comments and the comment volume information. For example, the selection unit 64 may select the video of the angle with the most comments or the video of the angle with the loudest audio comment volume as the video of the angle the user will view. Alternatively, the selection unit 64 may, for example, analyze the feature quantities of the movement of the subject being filmed included in the video of each angle displayed in the first region 110 to the fourth region 140 in the multi-angle video 100 shown in Figure 4, and determine the degree of excitement. In this case, the selection unit 64 may determine that the video in which the subject being filmed was moving most vigorously is the most exciting, and select that video as the video of the angle the user will view. Furthermore, the selection unit 64 may select the video of the angle the user will view based on the number of comments, the comment volume information, and the degree of excitement. If it is determined that angle selection information has been obtained (step S66; Yes), proceed to step S68. If it is not determined that angle selection information has been obtained (step S66; No), proceed to step S70.

[0098] The processes from step S68 to step S76 are the same as the processes from step S26 to step S34 shown in Figure 8, so their explanation will be omitted.

[0099] In the second embodiment, by displaying comment count information and comment volume information on the video included in the multi-angle video, it becomes easier for the user to select the video they want to watch.

[0100] [Third Embodiment] (Processing of Server Device) The processing flow of the server device according to the third embodiment will be explained using Figure 11. Figure 11 is a flowchart showing the processing flow of the server device according to the third embodiment.

[0101] The processes from steps S80 to S84 are the same as the processes from steps S10 to S14 shown in Figure 7, so their explanation will be omitted.

[0102] The voice comment generation unit 36 ​​generates voice information as superimposed information for outputting voice comments to the terminal device 14 based on the voice comment information acquired by the voice comment acquisition unit 32 (step S86). Specifically, the voice comment generation unit 36 ​​generates voice information including directional information. Then, the process proceeds to step S88.

[0103] The distribution unit 38 distributes multi-angle video, audio commentary information, and audio information to the terminal device 14 (step S88). Then, the process shown in Figure 9 is completed.

[0104] (Processing of the terminal device) The processing flow of the terminal device according to the third embodiment will be explained using Figure 12. Figure 12 is a flowchart showing the processing flow of the terminal device according to the third embodiment.

[0105] The acquisition unit 60 acquires multi-angle video, audio commentary information, and audio information from the server device 12 (step S100). Then, the process proceeds to step S102.

[0106] The process in step S102 is the same as the process in step S22 shown in Figure 8, so its explanation will be omitted.

[0107] The playback unit 62 outputs audio related to the audio comment from the audio output unit 46 based on the audio information acquired by the acquisition unit 60 (step S104). Specifically, the playback unit 62 adjusts the direction from which the audio comment is heard according to the video being viewed by the user, based on the audio information acquired by the acquisition unit 60, and outputs the audio comment from the audio output unit 46. In other words, the playback unit 62 controls the direction from which the audio comment is heard depending on the video being viewed by the user.

[0108] The processes from step S106 to step S116 are the same as the processes from step S24 to step S34 shown in Figure 8, so their explanation will be omitted.

[0109] In the third embodiment, the direction from which audio commentary is heard is controlled according to the video being viewed by the user among the multi-angle video. This allows the third embodiment to enhance the sense of realism and impact provided to the user viewing the video.

[0110] [Fourth Embodiment] (Processing of Server Device) The processing flow of the server device according to the fourth embodiment will be explained using Figure 13. Figure 13 is a flowchart showing the processing flow of the server device according to the fourth embodiment.

[0111] The processes from step S120 to step S124 are the same as the processes from step S10 to step S14 shown in Figure 7, so their explanation will be omitted.

[0112] The audio comment acquisition unit 32 determines whether or not there are multiple audio comment pieces of information (step S126). Specifically, the audio comment acquisition unit 32 determines whether or not multiple audio comment pieces of information have been acquired for the same angle identification information. If it is determined that there are multiple audio comment pieces of information (step S126; Yes), the audio comment acquisition unit 32 may further determine that there are multiple audio comment pieces of information based on the playback time at which the audio comments were acquired. In other words, even if the audio comments were acquired for the same angle identification information, if the playback time at which the audio comments were acquired is different, it may be determined that there are not multiple audio comment pieces of information. Proceed to step S128. If it is not determined that there are multiple audio comment pieces of information (step S126; No), proceed to step S132.

[0113] Step S126; If Yes, the voice comment generation unit 36 ​​generates both voice information and text information for displaying the voice comment as text, as superimposed information, based on the voice comment information acquired by the voice comment acquisition unit 32 (Step S128). Specifically, if there are multiple voice comments for the same angle of video, the voice comment generation unit 36 ​​generates at least one voice comment as voice information and converts the other voice comment information into text information and superimposes it onto the video of the same angle. The voice comment generation unit 36 ​​may, for example, generate voice information based on the oldest voice comment information, or it may generate voice information based on the newest voice comment information. The voice comment generation unit 36 ​​may also analyze the contents of multiple voice comments acquired in the same angle identification information using a language recognition model (not shown) and generate text information. For example, it generates voice information based on the voice comment information that has been analyzed to have the highest similarity to the other comments among the multiple voice comments. Then, the process proceeds to step S130.

[0114] The distribution unit 38 distributes multi-angle video, audio information, and text information to the terminal device 14 (step S130). Then, the process shown in Figure 13 is completed.

[0115] The processes in steps S132 and S134 are the same as the processes in steps S86 and S88 shown in Figure 11, respectively, so their explanation will be omitted.

[0116] (Processing of the terminal device) The processing flow of the terminal device according to the fourth embodiment will be explained using Figure 14. Figure 14 is a flowchart showing the processing flow of the terminal device according to the fourth embodiment.

[0117] The acquisition unit 60 acquires multi-angle video, audio commentary information, audio information, and text information from the server device 12 (step S150). Then, the processing proceeds to step S152.

[0118] The process in step S152 is the same as the process in step S22 shown in Figure 8, so its explanation will be omitted.

[0119] The playback unit 62 causes the audio output unit 46 to output audio related to the audio comment based on the audio information acquired by the acquisition unit 60 (step S154). Specifically, the playback unit 62 causes the audio output unit 46 to output at least one audio comment. Then, the process proceeds to step S156.

[0120] The playback unit 62 displays text related to the audio comment on the display unit 44 based on the text information acquired by the acquisition unit 60 (step S156). Specifically, the playback unit 62 overlays the text onto the multi-angle video. For example, the playback unit 62 displays the text so that it is overlaid on the video that the user is watching. At this time, the playback unit 62 may display the content of the audio comment as text, or it may display the text with effects. For example, the playback unit 62 may display the text in a speech bubble or in an onomatopoeic style used in comics. The playback unit 62 may change the color of the font displayed based on the content of the text.

[0121] When the playback unit 62 plays back the multi-angle video shown in Figure 4 using the first playback method, it may display only the text related to the audio comments in the video of the angle displayed in the fifth region 150. In this case, the playback unit 62 may, for example, superimpose the text related to the audio comments in the video of the angle displayed in the fifth region 150 onto the fifth region 150. In addition to this, the playback unit 62 may superimpose the text related to the audio comments in the video of the corresponding angles into the first to fourth regions 110 to 140. In this case, the playback unit 62 may make the font size of the text displayed in the fifth region 150 larger than the font size of the text displayed in the first to fourth regions 110 to 140. The playback unit 62 may also display more audio comments in the fifth region 150 than the number of audio comments displayed in the first to fourth regions 110 to 140.

[0122] When the playback unit 62 plays back the multi-angle video shown in Figure 5 using the second playback method, it may superimpose and display text related to the audio comments for the corresponding angles of the video in the first to fourth regions 210 to 240.

[0123] In other words, the playback unit 62 may display text related to the audio comment on the display unit 44 based on the playback method.

[0124] The processes from step S158 to step S168 are the same as the processes from step S24 to step S34 shown in Figure 8, so their explanation will be omitted.

[0125] In the fourth embodiment, the content of the audio commentary is displayed as text along with the video the user is watching. This makes it easier for the user to understand the content of the audio commentary, thus making it easier to select a video. Furthermore, this also makes it easier to distinguish between multiple angles in the fourth embodiment.

[0126] Up to this point, the distribution system 1 has been described assuming real-time distribution, but it is not limited to this. For example, the distribution system 1 may also allow archived viewing of content that has been distributed in real time. Also, the audio comment acquisition unit 32 does not necessarily have to acquire audio comments from the terminal device 14. For example, the distribution system 1 may be equipped with a microphone (not shown) for acquiring audio comments near the subject being filmed, and the audio comment acquisition unit 32 may acquire the audio from the microphone (not shown) as audio comments. In this case, the terminal device 14 does not necessarily have to be equipped with an audio acquisition unit 66.

[0127] Each component of the illustrated device is a functional concept and does not necessarily have to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions. Furthermore, this distribution and integration configuration may be performed dynamically.

[0128] While embodiments of the present disclosure have been described above, the present disclosure is not limited by the content of these embodiments. Furthermore, the aforementioned components include those that are readily conceivable to those skilled in the art, those that are substantially identical, and those that fall within the so-called equivalent range. Moreover, the aforementioned components can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the spirit of the embodiments described above.

[0129] The server equipment and distribution method disclosed herein can be used, for example, in a content distribution service over a network.

[0130] 1 Distribution System 10, 10-1, 10-2, 10-3, 10-4, 10-n Camera 12 Server Device 14, 14-1, 14-m Terminal Device 20, 48 Communication Unit 22, 50 Storage Unit 24, 52 Control Unit 30 Video Acquisition Unit 32 Audio Comment Acquisition Unit 34 Video Generation Unit 36 ​​Audio Comment Generation Unit 38 Distribution Unit 40 Input Unit 42 Microphone 44 Display Unit 46 Audio Output Unit 60 Acquisition Unit 62 Playback Unit 64 Selection Unit 66 Audio Acquisition Unit 68 Communication Control Unit

Claims

1. A server device comprising: a video generation unit that synthesizes multiple angle videos of a subject filmed from different directions into a single video to generate a multi-angle video; a voice comment acquisition unit that acquires voice comment information indicating voice comments made by users of multiple terminal devices when they view the multi-angle video; a voice comment generation unit that generates superimposed information to be superimposed on the multi-angle video based on the voice comment information acquired by the voice comment acquisition unit; and a distribution unit that distributes the multi-angle video, the voice comment information, and the superimposed information to the multiple terminal devices.

2. The server device according to claim 1, characterized in that the voice comment generation unit generates, based on the voice comment information acquired by the voice comment acquisition unit, comment count information indicating the number of voice comments and comment volume information indicating the volume of the voice comments as the superimposed information for each of the multiple angles of video included in the multi-angle video.

3. The server device according to claim 1, characterized in that the voice comment generation unit generates voice information as superimposed information that adjusts the direction in which the voice comment is heard, based on directional information corresponding to the video of multiple angles.

4. The server device according to claim 3, wherein the audio comment generation unit generates the audio information based on the arrangement of the multiple angles of the video included in the multi-angle video.

5. The server device according to claim 1, characterized in that, when there are multiple audio comments for video of the same angle, the audio comment generation unit generates audio information from at least one of the audio comments and text information from the other audio comments, both as superimposed information.

6. A distribution method comprising: generating a multi-angle video by combining multiple angle videos of a subject filmed from different directions into a single video; acquiring audio comment information from multiple terminal devices indicating audio comments made by users of the terminal devices when they viewed the multi-angle video; generating superimposed information to be superimposed on the multi-angle video based on the acquired audio comment information; and distributing the multi-angle video, the audio comment information, or the superimposed information to the multiple terminal devices.

Images