Information processing device, method, and program

Abstract

An information processing device according to one embodiment of the present invention comprises: a creation unit that creates content corresponding to conditions of a plurality of types of qualities which are defined for each user who receives provision of image or audio content; and a management unit that, when a first quality related to the provision of the created content changes so as not to correspond to the conditions, performs management such that the first quality corresponds to the conditions by causing the creation unit to create the content under conditions in which a second quality that is related to the provision of the created content and that is different from the first quality is reduced.

Description

Information processing apparatus, method and program

[0001] Embodiments of the present invention relate to an information processing apparatus, method and program.

[0002] In the field of content distribution processing, there are video services that aim to allow users to have a simulated experience of an environment different from the actual environment where the user is located, by using video content. Examples of such services include immersive games that use VR (Virtual Reality) goggles to give users the feeling of entering the game world, or simulated travel services that allow users to have a simulated travel experience while watching a high-definition TV monitor (see, for example, Patent Documents 1 and 2).

[0003] In these services, characters encountered by the user in the game or tour guides who conduct travel guidance appear. When displaying these characters in content, instead of directly displaying pre-photographed characters in the content, for example, rendering and displaying 3D computer graphics (3DCG) in accordance with the user's utterances or actions can achieve effects such as expressing the communication between the user and the characters, thereby further enhancing the sense of realism of the service.

[0004] Japanese Unexamined Patent Application Publication No. 2023-118163 Japanese Unexamined Patent Application Publication No. 2018-201150

[0005] However, there are various problems such as an increase in processing load when making 3DCG avatars high-definition, and the difficulty of reproducing natural movements exactly like those of real humans in 3DCG avatars. Therefore, the threshold for creating a service that is close to video of real persons is relatively high. As a result, there is a problem that services intended to enhance the sense of realism may conversely promote unnaturalness.

[0006] Furthermore, consideration must be given to various quality elements of video content, such as (A1) to (A4) below: (A1) When aiming to improve service quality in video distribution, reduce the delay between video and audio on the receiver side, i.e., reduce processing delay or communication delay. (A2) To improve the quality of interaction, reduce the delay in responses to conversations between the user and the avatar, i.e., reduce both processing delay and communication delay. (A3) Improve the quality of the video, i.e., increase the resolution. (A4) Improve the quality of the audio, i.e., use 3D audio, etc.

[0007] Addressing all of these factors would significantly increase the service load and lead to increased costs. Furthermore, conventional technologies generally do not consider all of these quality elements when improving video streaming services; most only improve one quality element at a time.

[0008] Based on cognitive psychology, an evaluation of the aforementioned video services revealed that the level of quality degradation that users tolerate varies from person to person, and the acceptable range of quality for each element also differs. Differences in individual user tolerance levels include, for example, the difference between "some people were bothered by video flickering but not so much by audio / video lag" and "some people were bothered by delays in dialogue responses but not so much by audio / video quality degradation." Furthermore, it was found that the acceptable range of degradation for each element also differs from person to person depending on the type of content.

[0009] This invention was made in view of the above circumstances, and its purpose is to provide an information processing device, method, and program that enable appropriate management of content quality.

[0010] An information processing device according to one aspect of the present invention includes a creation unit that creates content according to a plurality of quality conditions defined for each user who receives image or audio content, and a management unit that, when the first quality of the created content changes to a quality that does not conform to the conditions, causes the creation unit to create the content under conditions that are different from the first quality and lower the second quality of the created content, thereby managing the first quality to conform to the conditions.

[0011] An information processing method according to one aspect of the present invention is a method performed by an information processing device, comprising: a creation unit of the information processing device creating content according to a plurality of quality conditions defined for each user receiving image or audio content; and a management unit of the information processing device, when the first quality of the content provided changes to a quality that does not conform to the conditions, causing the creation unit to create the content under conditions that are different from the first quality and lower the second quality of the content provided, thereby managing the first quality to conform to the conditions.

[0012] According to the present invention, the quality of content can be appropriately managed.

[0013] Figure 1 is a diagram showing an example of the application of an information processing device according to one embodiment of the present invention. Figure 2 is a flowchart showing an example of the processing procedure by the user information acquisition unit. Figure 3 is a diagram showing an example of a user table. Figure 4 is a flowchart showing an example of the processing procedure by the content material collection unit. Figure 5 is a flowchart showing an example of the processing procedure by the content configuration management unit. Figure 6 is a block diagram showing an example of the hardware configuration of an information processing device according to one embodiment of the present invention.

[0014] Embodiments relating to this invention will be described below. Figure 1 is a diagram showing an example of application of an information processing device according to one embodiment of the present invention. As shown in Figure 1, the information processing device 100 according to one embodiment of the present invention comprises a user information acquisition unit 10, a content material collection unit 20, a content configuration management unit 30, a visual information configuration unit 40, an auditory information configuration unit 50, and a content configuration unit 60. The functions of each unit will be described later.

[0015] In this embodiment, an acceptable range for content quality elements can be set for each individual user, and content with the minimum necessary processing tailored to each individual can be presented. This allows for the presentation of content of a more suitable quality to each user while reducing costs compared to processing all elements.

[0016] Figure 2 is a flowchart showing an example of the processing procedure by the user information acquisition unit. Figure 3 is a diagram showing an example of a user table. The user information acquisition unit 10 acquires information on content having a configuration preferred by the user and stores information on the quality conditions for creating content having a configuration preferred by the user in a user table as shown in Figure 3 (S11). This user table can be stored, for example, in the internal memory of the user information acquisition unit 10.

[0017] This user table stores, for example, information about multiple quality conditions for the actual content preferred by a user corresponding to a user ID, such as the spatial resolution threshold (SrTh), temporal resolution threshold (TrTh), sound quality threshold (SqTh), image-to-sound synchronization threshold (SpsTh), and response synchronization threshold (RTh), for each user ID, which is the identification information of that user.

[0018] Each of these pieces of information includes two types of information: element values, which are the values ​​that make up each of the various quality elements preferred by the user, and the priority order among the different quality elements. In the example shown in Figure 3, the values ​​listed in the upper column for each user ID represent the various element values, and the values ​​listed in the lower column for each user ID represent the priority order.

[0019] Element values ​​may be pre-set by the service provider as average values, or they may be set by the user corresponding to the user ID prior to the provision of the service. Priority is indicated, for example, by a number between 0 and 10, with a lower value indicating a higher priority. This priority may be estimated and set by the service provider based on the user's behavior history from the last time they used the service.

[0020] If the user's behavior history from their previous use of the service is used, the content configuration unit 60 feeds back information about the content configuration to the user information acquisition unit 10. Alternatively, before the user uses the service, the service provider may present sample content to the user and conduct a questionnaire, and the results of this questionnaire may be used as the behavior history. Furthermore, as shown in Figure 3 for the "User ID: 03" column in the user table, only the priority may be set without setting the actual element value.

[0021] Next, the user information acquisition unit 10 sends the element values ​​stored in the user table to the content material collection unit 20, and the priority values ​​stored in the user table to the content configuration management unit 30 (S12).

[0022] Figure 4 is a flowchart showing an example of the processing procedure by the content material collection unit. The content material collection unit 20 determines whether the element value received from the user information acquisition unit 10 is related to vision, i.e., an element value of visual information; related to hearing, i.e., an element value of auditory information; an element value of information related to discrepancies between image and sound; or an element value of information related to response discrepancies.

[0023] If the element value received from the user information acquisition unit 10 is an element value of visual information, i.e., at least one of the element values ​​of the spatial resolution threshold of the image and the temporal resolution threshold of the image described above, the content material collection unit 20 passes this element value to the visual information configuration unit 40. If the element value received from the user information acquisition unit 10 is an element value of auditory information, i.e., an element value of the sound quality threshold described above, the content material collection unit 20 passes this element value to the auditory information configuration unit 50.

[0024] Furthermore, if the element value received from the user information acquisition unit 10 is at least one of the element values ​​related to the discrepancy between video and sound, i.e., the element value of the video-to-sound discrepancy threshold described above, and the element value related to the discrepancy between responses, i.e., the element value of the response discrepancy threshold described above, the content material collection unit 20 passes this element value to the content composition unit 60 (S21).

[0025] Simultaneously with the process of passing element values ​​in S21, the content material collection unit 20 commands the visual information structuring unit 40 to start processing visual information based on the element values ​​of the visual information passed as described above, commands the auditory information structuring unit 50 to start processing auditory information based on the element values ​​of the auditory information passed as described above, and commands the content structuring unit 60 to start processing based on the element values ​​passed as described above (S22).

[0026] When the visual information structuring unit 40 and the auditory information structuring unit 50 receive element values ​​from the content material collection unit 20, they create content that conforms to the specifications requested by the user based on these element values. For example, when structuring video content suitable for the user "User ID: 01" in the user table shown in Figure 3, the visual information structuring unit 40 receives from the content material collection unit 20 that the element value for the spatial resolution threshold of the image related to "User ID: 01" is "None", and the element value for the temporal resolution threshold of the image related to "User ID: 01" is "60 fps", and prepares video content materials that conform to the following conditions (B1) and (B2): (B1) The spatial resolution of the image is arbitrary. (B2) The temporal resolution of the image is 60 fps or higher.

[0027] Furthermore, for example, when configuring audio content suitable for the user "User ID: 01" in the user table shown in Figure 3, the auditory information configuration unit 50 prepares audio content materials that meet the condition "audio quality is 320 kbps or higher" because the element value of the sound quality threshold related to "User ID: 01" is "320 kbps".

[0028] At this time, the processing by the visual information structuring unit 40 and the processing by the auditory information structuring unit 50 are monitored by the content configuration management unit 30. Details of the processing by the content configuration management unit 30 will be described later.

[0029] The content materials processed by the visual information structuring unit 40 and the auditory information structuring unit 50 are passed to the content structuring unit 60. The content structuring unit 60 uses the content materials received from the visual information structuring unit 40 and the auditory information structuring unit 50, and generates moving image content by synthesizing images and sounds according to the element values ​​received from the content material collection unit 20.

[0030] For example, when configuring video content suitable for the user "User ID: 01" in the user table shown in Figure 3, the content configuration unit 60, knowing that the element value of the image-to-sound delay threshold related to "User ID: 01" is "None" and the element value of the response delay threshold related to "User ID: 01" is "30 ms", prepares video content material that conforms to the following conditions (C1) and (C2), and generates video content according to this material. (C1) The delay between image and sound is arbitrary. (C2) The response delay is within 30 ms.

[0031] The generated content is displayed on the information processing device 100 or on an external content display device (not shown). Furthermore, the content information configured in the content configuration unit 60 may be sent to the user information acquisition unit 10 and stored in the user table for future use.

[0032] Figure 5 is a flowchart showing an example of the processing procedure by the Content Configuration Management Unit. The Content Configuration Management Unit 30 receives the priority on the user table shown in Figure 3 from the User Information Acquisition Unit 10 (S31), and based on this received priority on the user table shown in Figure 3, it monitors the operation of the Visual Information Configuration Unit 40, the Auditory Information Configuration Unit 50, and the Content Configuration Unit 60, which are the content configuration processes (S32). Here, the operation is explained using the example of generating content suitable for the user with "User ID: 02" in the user table shown in Figure 3.

[0033] The video content for user "User ID: 02" is generated by processing the content in each section according to the specifications (D1) to (D5) below. (D1) Spatial resolution of the image is 1920 x 1080 pixels or higher. (D2) Temporal resolution of the image is 30 fps or higher. (D3) Audio information per second (kbps) is arbitrary. (D4) Video-to-audio lag is within 30 ms. (D5) Response lag is within 90 ms.

[0034] The content configuration management unit 30 monitors changes in the process being monitored in S32, and if there are no deviations from the specifications of each element (No. in S33), it does not perform any special processing and allows the processing related to content configuration to continue to the visual information configuration unit 40, auditory information configuration unit 50, and content configuration unit 60 (S34).

[0035] Here, let's assume that the spatial resolution of the image is too high for the processing specifications of the visual information configuration unit 40, causing the image processing to take too long, and that the delay between the image and sound in the content configuration unit 60 changes to 35 ms, exceeding the 30 ms shown in (D4) above. In this case, the content configuration management unit 30 detects that there is a deviation from the specifications of the element being monitored (Yes in S33) and commands each unit to change its processing (S35).

[0036] In this case, the content configuration management unit 30 uses the priority information received from the user information acquisition unit 10. In the column for "User ID: 02" in the user table shown in Figure 3, the priority of the image-to-sound misalignment threshold (SpsTh) related to the image-to-sound misalignment that deviated from the specifications as described above is 4.49, and when comparing the priority with other elements related to the same user ID, it is expressed as follows (1): Image temporal resolution threshold (TrTh) < (Image-to-sound misalignment threshold (SpsTh) < Response misalignment threshold (RTh), Image spatial resolution threshold (SrTh), Sound quality threshold (SqTh) ... (1) From the results of this comparison, the only element that should be processed with priority over the image-to-sound misalignment, that is, the element that requires a change in processing to each unit, is the image temporal resolution, and the response misalignment, image spatial resolution, and sound quality are not elements that should be processed with priority over the image-to-sound misalignment.

[0037] Therefore, for example, the content configuration management unit 30 issues a command to the visual information configuration unit 40 to change the processing in order to resolve the discrepancy between video and audio. In response to this command, the visual information configuration unit 40 creates video content by reducing the element value of the spatial resolution threshold (SrTh) of the image related to "User ID: 02" from, for example, an element value that satisfies "1920 x 1080 pic or higher" corresponding to the current spatial resolution threshold of the image to an element value that satisfies "1280 x 720 pic or higher," thereby resolving the reduction in image processing speed.

[0038] Furthermore, if the aforementioned discrepancy between video and audio is caused by the audio processing taking too much time, the content configuration management unit 30 may issue an instruction to the auditory information configuration unit 50 to change the processing in order to resolve the discrepancy between video and audio. In response to this instruction, the auditory information configuration unit 50 may resolve the reduction in audio processing speed by creating audio content with a reduced element value of the sound quality threshold (SqTh) related to "User ID: 02".

[0039] As shown in Figure 3, the priority of content required by the user with "User ID: 02" has been measured in advance. Therefore, as mentioned above, reducing the spatial resolution of the images will not pose a major problem for the user with "User ID: 02".

[0040] Furthermore, when configuring content for a user, such as the user with "User ID: 03" in the user table shown in Figure 3, where no element values ​​are set and only priority is defined, the content configuration management unit 30 calculates which processes to prioritize and to what extent based on the processing specifications of each part of the information processing device 100, manages the content configuration, and commands each part to change its processing.

[0041] In the user table shown in Figure 3, for user "User ID: 03", the priority order from lowest to highest is sound quality threshold, image-to-sound delay threshold, response delay, spatial resolution of the image, and temporal resolution of the image. Therefore, the content configuration management unit 30 allocates the most resources to audio processing, followed by image-to-sound synthesis, response delay, spatial resolution of the image, and temporal resolution of the image, and manages the content configuration by allocating resources in that order, and commands each unit to change its processing.

[0042] Figure 6 is a block diagram showing an example of the hardware configuration of an information processing device according to one embodiment of the present invention. In the example shown in Figure 6, the information processing device 100 according to the above embodiment is composed of, for example, a server computer or a personal computer, and has a hardware processor 111A such as a CPU (Central Processing Unit). A program memory 111B, a data memory 112, an input / output interface 113, and a communication interface 114 are connected to this hardware processor 111A via a bus 115.

[0043] The communication interface 114 includes, for example, one or more wireless communication interface units, enabling the transmission and reception of information with the communication network. As the wireless interface, for example, an interface employing a low-power wireless data communication standard such as a wireless LAN (Local Area Network) is used.

[0044] To the input / output interface 113, an input device 200 and an output device 300, which are attached to the information processing apparatus 100 and used by a user or the like, are connected.

[0045] The input / output interface 113 can acquire operation data input by a user or the like through the input device 200 such as a keyboard, a touch panel, or a touchpad, and perform processing of outputting output data to the output device 300 including a display device using liquid crystal or organic EL (Electro Luminescence) or the like to cause the output device 300 to display the output data. Note that a device built in the information processing apparatus 100 may be used as the input device 200 and the output device 300, or an input device and an output device of another information terminal that can communicate with the information processing apparatus 100 via a network may also be used.

[0046] The program memory 111B is a non-transitory tangible storage medium, for example, formed by combining a non-volatile memory capable of random writing and reading such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive) and a non-volatile memory such as a ROM (Read Only Memory), and can store programs required for executing various control processes and the like according to an embodiment.

[0047] The data memory 112 is a tangible storage medium, for example, formed by combining the aforementioned non-volatile memory and a volatile memory such as a RAM (Random Access Memory), and can be used for storing various data or information acquired and created in the process of performing various processes.

[0048] The information processing apparatus 100 according to an embodiment of the present invention can be configured as a data processing apparatus including each part of the information processing apparatus 100 as a processing function unit implemented by software.

[0049] The storage devices used as work memories by each part of the information processing device 100 may be configured using the data memory 112 shown in Figure 6. However, the storage areas configured by these storage devices are not essential to the information processing device 100, and may be, for example, areas provided in external storage media such as USB (Universal Serial Bus) memory, or storage devices such as database servers located in the cloud.

[0050] All of the above processing functions can be implemented by having the hardware processor 111A read and execute a program stored in the program memory 111B. Some or all of these processing functions may be implemented in various other forms, including application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs).

[0051] Furthermore, the methods described in each embodiment can be stored as programs (software means) that can be executed by a computer on recording media such as magnetic disks (floppy disks, hard disks, etc.), optical disks (CD-ROMs, DVDs, MOs, etc.), and semiconductor memories (ROMs, RAMs, flash memories, etc.), and can also be transmitted and distributed via communication media. The programs stored on the media also include configuration programs that configure the computer to run software means (including not only the execution program but also tables or data structures). The computer implementing this device reads the program recorded on the recording media and, if necessary, constructs the software means using the configuration program, and executes the above-described processes by controlling the operation of this software means. Note that the recording media referred to in this specification are not limited to those for distribution, but also include storage media such as magnetic disks or semiconductor memories provided inside the computer or in devices connected via a network.

[0052] It should be noted that the present invention is not limited to the embodiments described above, and can be modified in various ways during implementation without departing from its essence. Furthermore, each embodiment may be combined as appropriate, and in that case, the combined effects can be obtained. Moreover, the above embodiments include various inventions, and various inventions can be extracted by selecting combinations from the multiple constituent elements disclosed. For example, if the problem can be solved and effects obtained even if some constituent elements are deleted from all the constituent elements shown in the embodiment, then the configuration with these deleted constituent elements can be extracted as an invention.

[0053] 100... Information processing device 10... User information acquisition unit 20... Content material collection unit 30... Content configuration management unit 40... Visual information configuration unit 50... Auditory information configuration unit 60... Content configuration unit

Claims

1. An information processing device comprising: a creation unit that creates content according to multiple quality conditions defined for each user receiving image or audio content; and a management unit that, when the first quality of the content provided changes to a quality that does not conform to the conditions, causes the creation unit to create the content under conditions that are different from the first quality and lower the second quality of the content provided, thereby managing the first quality to conform to the conditions.

2. For each of the multiple quality conditions, a priority is set for selecting a quality to be reduced by the management unit, and when the first quality relating to the provision of the created content changes to a quality that does not conform to the conditions, the management unit selects a quality that is different from the first quality and has a higher priority set for that first quality and reduces it so that the first quality conforms to the conditions.

3. An information processing method performed by an information processing device, comprising: a creation unit of the information processing device creating content according to multiple quality conditions defined for each user receiving image or audio content; and a management unit of the information processing device, when the first quality of the content provided changes to a quality that does not conform to the conditions, causing the creation unit to create the content under conditions that are different from the first quality and reduce the second quality of the content provided, thereby managing the first quality to conform to the conditions.

4. An information processing program that causes a processor to function as a component of the information processing apparatus described in claim 1 or 2.

Images