Method and system for providing video conference service including artificial intelligence-based speech interpretation function

Abstract

Disclosed are a method and system for providing a speech conference service including an artificial intelligence-based speech interpretation function. According to one embodiment, the method for providing a video conference service may comprise the steps of: setting two or more interpretation bots participating as virtual participants in a video conference service; interpreting a speech of a first language input by participants of the video conference service into a speech of at least one other language different from the first language through processes of speech recognition, translation, and synthetic speech generation between the two or more interpretation bots and artificial intelligence; and providing the interpreted speech.

Description

Method and system for providing a video conferencing service including an AI-based voice interpretation function

[0001] The following description relates to a method and system for providing a video conferencing service including an AI-based voice interpretation function.

[0002] Conventional technology exists for displaying translated subtitles in real time during online video conferences by converting voice input in a first language into text using STT (Speech-To-Text) and then converting the converted text back into text in a second language. However, there is a problem in that these translated subtitles are displayed alongside the video or screen-shared materials of the participants, making it difficult for users to concentrate on the meeting.

[0003] In addition, there are prior art technologies that generate meeting minutes using STT on recorded audio, or generate minutes summaries or provide Q&A functions through a Large Language Model (LLM) by utilizing the recorded minutes as context after the meeting ends. However, these prior art technologies have limitations in providing AI-based functions using the content of an online video conference in real time while the meeting is in progress.

[0004] A method and system for providing a video conferencing service including an artificial intelligence-based voice interpretation function are provided.

[0005] A method for providing a video conferencing service in a system for providing a video conferencing service implemented by at least one computer device, wherein the at least one computer device includes at least one processor, and the method for providing a video conferencing service comprises: a step of setting up two or more interpreter bots participating as virtual participants in a video conferencing service by the at least one processor; a step of interpreting a voice of a first language input by the participants of the video conferencing service into a voice of at least one other language different from the first language through a process of voice recognition, translation, and synthetic voice generation between the two or more interpreter bots and artificial intelligence by the at least one processor; and a step of providing the interpreted voice by the at least one processor.

[0006] According to one aspect, the step of setting up the two or more interpretation bots may be characterized by including the step of setting up the output language of each of the two or more interpretation bots and at least one of the voices of each of the participants by using at least one of the names of the participants, the voices of the participants, the language used by the participants, and the countries of the participants.

[0007] According to another aspect, the interpreting step may be characterized by comprising: a step of generating text of the first language by recognizing the voice of the first language through the artificial intelligence and a first interpreting bot implemented to process the voice of the first language among the two or more interpreting bots; a step of translating the text of the first language into text of the second language through the artificial intelligence and a second interpreting bot implemented to process the voice of the second language among the two or more interpreting bots; and a step of converting the text of the second language into the voice of the second language through the second interpreting bot and the artificial intelligence.

[0008] According to another aspect, the interpreting step may further include: a step of translating text of a first language into text of a third language through artificial intelligence and a third interpreting bot implemented to process the voice of a third language among the two or more interpreting bots; and a step of converting text of a third language into voice of a third language through the third interpreting bot and artificial intelligence.

[0009] According to another aspect, the step of providing the interpreted voice may be characterized by providing a voice of the second language so that the voice of the second language is output through a channel for the second language, providing a voice of the third language so that the voice of the third language is output through a channel for the third language, outputting a voice of the second language interpreted from the voice of the first language to a participant who has selected a channel for the second language, and outputting a voice of the third language interpreted from the voice of the first language to a participant who has selected a channel for the third language.

[0010] According to another aspect, the step of generating the text may be characterized by requesting Speech-To-Text (STT) conversion of the speech of the first language to the artificial intelligence through the first interpreter bot to recognize the speech of the first language through the artificial intelligence and generate text of the first language, and the step of translating may be characterized by requesting translation of the text of the first language to text of the second language to the artificial intelligence through the second interpreter bot to translate the text of the first language into text of the second language through the artificial intelligence, and the step of converting may be characterized by requesting Text-To-Speech (TTS) conversion of the text of the second language to the artificial intelligence through the second interpreter bot to convert the text of the second language into speech of the second language through the artificial intelligence.

[0011] According to another aspect, the speech recognition is performed through Speech-To-Text (STT) conversion using the artificial intelligence, and the interpreting step may be characterized by including: a step of transmitting the speech of a first language to the artificial intelligence in real time for the STT conversion to receive the STT result, and determining whether the sentence ends according to the received STT result; and a step of performing the translation and speech synthesis using the artificial intelligence on the sentence-unit text detected according to whether the sentence ends.

[0012] According to another aspect, the interpreting step may be characterized by including a step of inputting the result of the speech recognition using the artificial intelligence into a Large Language Model (LLM) along with at least one of the meeting materials and the participants' speech glossary to correct the result of the speech recognition.

[0013] According to another aspect, the above-mentioned speech glossary may be characterized by including pairs of a participant's voice utterance and text corresponding to the voice utterance.

[0014] According to another aspect, the method for providing the video conferencing service may further include the step of providing directly requested data or indirectly requested data in real time based on meeting content recorded from voice being input in real time through the video conferencing service by the at least one processor.

[0015] According to another aspect, the directly requested data may include data requested by voice input through the video conferencing service, and the indirectly requested data may include answers generated based on the artificial intelligence for questions that have no answers from other participants.

[0016] According to another aspect, the step of providing the data in real time may be characterized by providing the data in at least one form among a Markdown document, a LaTeX document, executable code, and an image.

[0017] According to another aspect, the interpreting step may be characterized by including: a step of recognizing emotions through facial expressions of a person in a video transmitted and received through the video conferencing service; and a step of correcting at least one of the result of the voice recognition and the result of the translation based on the recognized emotions.

[0018] According to another aspect, the method for providing the video conferencing service may further include the step of recognizing text of a document appearing on a screen shared with the participants through the video conferencing service by the at least one processor using Optical Character Recognition (OCR) technology; and the step of providing the recognized text or a real-time translation result of the recognized text into another language by the at least one processor.

[0019] A computer program stored on a computer-readable recording medium is provided to be combined with a computer device to execute the above method on the computer device.

[0020] A computer-readable recording medium is provided on which a computer program for executing the above method is recorded on a computer device.

[0021] A computer device is provided for a video conferencing service providing system implemented by at least one computer device, wherein the at least one computer device includes at least one processor, and the at least one processor sets up two or more interpreter bots participating as virtual participants in the video conferencing service, interprets a voice of a first language input by the participants of the video conferencing service into a voice of at least one other language different from the first language through a process of voice recognition, translation, and synthetic voice generation between the two or more interpreter bots and artificial intelligence, and provides the interpreted voice.

[0022] A method and system for providing a video conferencing service including an AI-based voice interpretation function can be provided.

[0023] FIG. 1 is a drawing illustrating an example of a network environment according to an embodiment of the present invention.

[0024] FIG. 2 is a block diagram illustrating an example of a computer device according to an embodiment of the present invention.

[0025] FIG. 3 is a diagram illustrating an example of a general view of a video conferencing service providing system in one embodiment of the present invention.

[0026] FIG. 4 is a diagram illustrating an example of a process for processing a user's speech in an embodiment of the present invention.

[0027] Figure 5 is a diagram illustrating an example of a unidirectional interpretation process.

[0028] FIG. 6 is a diagram illustrating an example of a two-way interpretation process according to an embodiment of the present invention.

[0029] FIG. 7 is a drawing illustrating an example of text storage and utilization in an embodiment of the present invention.

[0030] FIGS. 8 and 9 are drawings illustrating an example of an implementation of a function providing system in an embodiment of the present invention.

[0031] FIG. 10 is a flowchart illustrating an example of a method for providing a video conferencing service in one embodiment of the present invention.

[0032] Hereinafter, embodiments will be described in detail with reference to the attached drawings.

[0033] A video conferencing service providing system according to embodiments of the present invention may be implemented by at least one computer device. In this case, a computer program according to an embodiment of the present invention may be installed and run on at least one computer device, and at least one computer device may perform a video conferencing service providing method according to embodiments of the present invention under the control of the run computer program. The above-described computer program may be stored on a computer-readable recording medium to be combined with at least one computer device to execute the video conferencing service providing method on a computer.

[0034] FIG. 1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention. The network environment of FIG. 1 illustrates an example including a plurality of electronic devices (110, 120, 130, 140), a plurality of servers (150, 160), and a network (170). FIG. 1 is an example for explaining the invention, and the number of electronic devices or servers is not limited to that shown in FIG. 1. Furthermore, the network environment of FIG. 1 is merely an example of one of the environments applicable to the present embodiments, and the environments applicable to the present embodiments are not limited to the network environment of FIG. 1.

[0035] Multiple electronic devices (110, 120, 130, 140) may be fixed terminals or mobile terminals implemented as computer devices. Examples of multiple electronic devices (110, 120, 130, 140) include smartphones, mobile phones, navigation systems, computers, laptops, digital broadcasting terminals, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), tablet PCs, etc. For example, FIG. 1 shows the shape of a smartphone as an example of an electronic device (110), but in embodiments of the present invention, the electronic device (110) may substantially refer to one of various physical computer devices capable of communicating with other electronic devices (120, 130, 140) and / or servers (150, 160) via a network (170) using a wireless or wired communication method.

[0036] The communication method is not limited and may include not only communication methods utilizing communication networks (e.g., mobile communication networks, wired internet, wireless internet, broadcasting networks) that the network (170) may include, but also short-range wireless communication between devices. For example, the network (170) may include any one or more networks such as a PAN (personal area network), LAN (local area network), CAN (campus area network), MAN (metropolitan area network), WAN (wide area network), BBN (broadband network), and the Internet. Additionally, the network (170) may include any one or more network topologies such as a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or hierarchical network, but is not limited thereto.

[0037] Each of the servers (150, 160) may be implemented as a computer device or multiple computer devices that communicate with multiple electronic devices (110, 120, 130, 140) through a network (170) to provide commands, code, files, content, services, etc. For example, the server (150) may be a system that provides services to multiple electronic devices (110, 120, 130, 140) connected through the network (170).

[0038] FIG. 2 is a block diagram illustrating an example of a computer device according to an embodiment of the present invention. Each of the plurality of electronic devices (110, 120, 130, 140) or servers (150, 160) described above can be implemented by the computer device (200) illustrated in FIG. 2.

[0039] As illustrated in FIG. 2, such a computer device (200) may include memory (210), a processor (220), a communication interface (230), and an input / output interface (240). The memory (210) is a computer-readable recording medium and may include a non-perishable mass storage device such as RAM (random access memory), ROM (read only memory), and a disk drive. Here, a non-perishable mass storage device such as a ROM and a disk drive may be included in the computer device (200) as a separate permanent storage device distinct from the memory (210). Additionally, an operating system and at least one program code may be stored in the memory (210). These software components may be loaded into the memory (210) from a computer-readable recording medium separate from the memory (210). This separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, or a memory card. In another embodiment, software components may be loaded into memory (210) via a communication interface (230) rather than a computer-readable recording medium. For example, software components may be loaded into memory (210) of a computer device (200) based on a computer program installed by files received through a network (170).

[0040] The processor (220) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor (220) via memory (210) or a communication interface (230). For example, the processor (220) may be configured to execute instructions received according to program code stored in a recording device such as memory (210).

[0041] The communication interface (230) may provide a function for the computer device (200) to communicate with other devices (e.g., storage devices described above) through the network (170). For example, requests, commands, data, files, etc. generated by the processor (220) of the computer device (200) according to program code stored in a recording device such as memory (210) may be transmitted to other devices through the network (170) under the control of the communication interface (230). Conversely, signals, commands, data, files, etc. from other devices may be received by the computer device (200) through the communication interface (230) of the computer device (200) via the network (170). Signals, commands, data, etc. received through the communication interface (230) may be transmitted to the processor (220) or memory (210), and files, etc. may be stored in a storage medium (the permanent storage device described above) that the computer device (200) may further include.

[0042] The input / output interface (240) may be a means for interfacing with an input / output device (250). For example, the input device may include a device such as a microphone, keyboard, or mouse, and the output device may include a device such as a display or speaker. As another example, the input / output interface (240) may be a means for interfacing with a device in which the functions for input and output are integrated into one, such as a touchscreen. At least one of the input / output devices (250) may be configured as a single device with the computer device (200). For example, it may be implemented in a form in which a touchscreen, microphone, speaker, etc., are included in the computer device (200), such as in a smartphone.

[0043] Additionally, in other embodiments, the computer device (200) may include fewer or more components than the components of FIG. 2. However, it is not necessary to clearly illustrate most of the prior art components. For example, the computer device (200) may be implemented to include at least some of the input / output devices (250) described above, or may include other components such as a transceiver, a database, etc.

[0044] FIG. 3 is a diagram illustrating an example of a general view of a video conferencing service providing system in an embodiment of the present invention. The video conferencing service providing system (300) according to the present embodiment may include a video conferencing system (310) and a function providing system (320). At this time, each of the video conferencing system (310) and the function providing system (320) may be implemented by at least one computer device, and may be implemented and operated by the same entity or by different entities. For example, a user (330) may receive a video conferencing service provided by the video conferencing system (310) through a user terminal (340). At this time, the function providing system (320) may communicate in conjunction with the video conferencing system (310) and provide various artificial intelligence-based functions to the user of the video conferencing system (310).

[0045] The function providing system (320) may utilize internal artificial intelligence, but may also integrate an externally provided AI model into its application based on API (Application Programming Interface) access. The embodiment of FIG. 3 illustrates an example of utilizing external artificial intelligence (350) based on API access.

[0046] This function providing system (320) can provide voice interpretation functions based on artificial intelligence (350) to the video conferencing system (310). In other words, the user (330) can receive voice interpretation functions from the video conferencing service provided by the video conferencing system (310) through the user terminal (340). At this time, an interpretation bot can participate as a participant in the video conferencing service to provide interpretation functions for the users' voices. Additionally, the function providing system (320) can execute a pipeline of voice recognition, translation, and voice synthesis with low latency to interpret the users' voices into another language and output them through the interpretation bot. At this time, the function providing system (320) can determine the language and gender of the voice output by the interpreter bot based on the user's (330) name (e.g., the name in the video conferencing service provided by the video conferencing system (310)), the user's (330) voice, the language used by the user (330), and / or the country set for the user (330). At this time, even for the same interpreter bot, the voice output may be set to a voice specific to each user depending on which user the voice output by the interpreter bot is delivered to.

[0047] FIG. 4 is a diagram illustrating an example of a process for processing user speech in an embodiment of the present invention. The embodiment of FIG. 4 describes an example of a process for interpreting user speech (421) in a first language into a second language. A user client (410) can transmit audio containing user voice (422) input according to user speech (421) to a video conferencing system (310). At this time, the video conferencing system (310) can transmit audio containing user voice (422) to a function providing system (320) through an audio callback (423).

[0048] The function providing system (320) can transmit a Speech-To-Text (STT) request (424) to the artificial intelligence (350) for converting the user voice (422) into text, and can receive text corresponding to the user voice (422) from the artificial intelligence (350) as an STT response (425). At this time, the text may include content expressed in first characters corresponding to the first language. Subsequently, the function providing system (320) can transmit a translation request (426) to the artificial intelligence (350) for converting the content of the text expressed in first characters into second characters corresponding to the second language, and can receive text including content expressed in second characters from the artificial intelligence (350) as a translation response (427). Additionally, the function providing system (320) can transmit a Text-To-Speech (TTS) request (428) to the artificial intelligence (350) for converting text containing content expressed in the second character into audio containing the voice of the second language, and can receive audio containing the voice of the second language from the artificial intelligence (350) as a TTS response (429).

[0049] In this case, the function providing system (320) can transmit a translation message (430) generated based on text containing content expressed in a second character to a user client (410) through a video conferencing system (310), and can transmit audio (431) generated based on audio containing voice of a second language to a user client (410) through a video conferencing system (310).

[0050] The user client (410) can output a translated message (430) corresponding to a user utterance (421) in the first language and generated audio (431) as an interpreted voice. At this time, the interpreted voice can be output through a channel for the second language, and each participant of the video conferencing service can select the channel through which the voice is output. In this case, participants of the video conferencing service who select the channel for the second language can hear the voice in the second language corresponding to the user utterance (421) in the first language.

[0051] Meanwhile, the function providing system (320) can provide a bidirectional interpretation bot based on artificial intelligence (350) for simultaneous interpretation of n (n is a natural number greater than or equal to 2) languages. For example, the interpretation bot may have a structure capable of bidirectional interpretation rather than unidirectional interpretation, such as from Korean to Japanese or from Japanese to Korean.

[0052] In video conferencing services, bots can be created and invited instead of humans, and these bots can receive and send audio just like regular human participants. When an interpreter bot is designated as an interpreter, it can be configured to listen only to audio from a specific language channel and send audio only to that specific language channel. For example, an interpreter bot designated as [Korean (KO) → English (EN)] can selectively listen only to the audio of other participants who have designated themselves as Korean speakers, and can transmit its (interpreted) audio only to participants who have designated themselves as English speakers. In this case, it is difficult for the interpreter bot to switch interpretation channels in real time, and a single participant cannot transmit different audio to multiple channels simultaneously. For instance, even if there is a program capable of simultaneously translating and synthesizing Korean into English and Japanese, if a bot is created as a participant in the video conferencing service, it cannot send English and Japanese audio to the English channel at the same time, and similarly, it cannot send English and Japanese audio to the Japanese channel at the same time. Therefore, the bot is only capable of sequential transmission, such as sending voice to the English channel first and then switching to the Japanese channel to send voice. Consequently, in situations where three or more languages ​​need to be interpreted simultaneously, multiple interpreter bot participants must be added to enable simultaneous interpretation for all languages.

[0053] The generally accepted structure of interpretation involves assigning interpreters who start from a specific language and interpret into a specific language. For example, in the case of [Korean (KO) ↔ English (EN)] bidirectional interpretation, a total of two interpreters are required: (1) an interpreter who interprets Korean into English, and (2) an interpreter who interprets English into Korean. If interpretation is performed sequentially after speech in each language, one person may take turns interpreting for two languages; however, if speech may occur simultaneously, a person who listens to Korean and a person who listens to English may always be required. However, in this case, if the number of languages ​​increases to three, six interpreters are required, and if it increases to four, ten interpreters are required.

[0054] FIG. 5 is a diagram illustrating an example of a unidirectional interpretation process, and FIG. 6 is a diagram illustrating an example of a bidirectional interpretation process according to an embodiment of the present invention.

[0055] In the embodiment of FIG. 5, an example is shown in which six interpretation bots—[KO→EN] bot (521), [KO→JA] bot (522), [EN→KO] bot (523), [EN→JA] bot (524), [JA→KO] bot (525), and [JA→EN] bot (526)—are used to enable interpretation between three languages, Korean (KO), English (EN), and Japanese (JA), for the participants (510). As such, when simultaneous interpretation of n languages ​​is required, n 2 Interpretation bots proportional to it are required.

[0056] In the embodiment of FIG. 6, an example is shown in which three interpretation bots—[KO] bot (610), [JA] bot (620), and [EN] bot (630)—are used in a function providing system (320) to enable interpretation between three languages: Korean (KO), English (EN), and Japanese (JA). Each interpretation bot can convert the speech of the language it is responsible for into text corresponding to the language it is responsible for through artificial intelligence (350) when speech is input, and store it in a shared storage called a queue (640). Additionally, each interpretation bot checks the text stored in the queue (640), and if there is text expressed in characters of a language different from its own language (other characters), it can perform translation and speech synthesis into the language it is responsible for through artificial intelligence (350). Finally, each interpretation bot can output only the speech of the language it is responsible for.

[0057] In the embodiment of FIG. 6, when someone among the participants (510) who speaks Korean makes the utterance "Hello," (1) recorded audio is transmitted to the video conferencing system (310), and (2) the video conferencing system (310) can transmit the recorded audio to the [KO] bot (610) to translate Korean into another language. All interpreter bots can collect voice from the video conferencing system (310) using an SDK (Software Development Kit), but if the voice is not of the language they are responsible for, they can ignore the voice without processing it. In other words, the [KO] bot (610), the [JA] bot (620), and the [EN] bot (630) can all collect audio containing Korean voice from the video conferencing system (310), but the [JA] bot (620) and the [EN] bot (630) can ignore audio containing Korean voice.

[0058] (3) [KO] bot (610) can convert Korean speech into text using artificial intelligence (350). An example has been described above in which an STT request is sent to the artificial intelligence (350) to convert speech into text using STT, and an STT response is received. (4) The converted text can be stored in a queue (640). Here, the text stored in the queue (640) may include Korean text. As previously described, [JA] bot (620) and [EN] bot (630) can each continuously check for the existence of text of other characters in the queue (640), (5) and when it is confirmed that text of other characters (Korean text) is stored in the queue (640), the Korean text can be retrieved, and (6) translation and speech synthesis of the Korean text into their respective languages ​​can be performed using artificial intelligence (350). For example, the [JA] bot (620) can translate Korean text into Japanese characters through artificial intelligence (350) and then synthesize the text of the translated characters into Japanese voice (voice synthesis using TTS) through artificial intelligence (350), and the [EN] bot (630) can translate Korean text into English characters through artificial intelligence (350) and then synthesize the text of the translated characters into English voice through artificial intelligence (350). Afterwards, (7) the [JA] bot (620) and the [EN] bot (630) can each transmit the synthesized voice to a video conferencing system (310), and (8) the video conferencing system (310) can provide the transmitted synthesized voice to the participants (510) through the respective language channels. For example, the Japanese synthesized voice provided by the [JA] bot (620) can be provided to participants who have joined the Japanese channel through the Japanese channel in the video conferencing system (310), and the English synthesized voice provided by the [EN] bot (630) can be provided to participants who have joined the English channel through the English channel in the video conferencing system (310).

[0059] FIG. 7 is a diagram illustrating an example of text storage and utilization in an embodiment of the present invention. A function providing system (320) may include a web API server (710) that records text converted from speech using STT through artificial intelligence (350), or may be linked with a web API server (710) implemented separately from the function providing system (320). FIG. 7 illustrates an example in which the function providing system (320) includes a web API server (710). For example, an interpreter bot (720) provided by the function providing system (320) may transmit the STT result obtained through artificial intelligence (350) to the web API server (710), and the web API server (710) may record this as a meeting minutes in a database (730). The web API server (710) can generate a meeting summary by transmitting the contents of the meeting minutes recorded in the database (730) to the artificial intelligence (350), or can generate and provide answers to the participants' questions regarding the meeting content using the artificial intelligence (350). Additionally, the web API server (710) can provide search results for the participants' searches regarding the meeting content.

[0060] In addition, the function providing system (320) can improve real-time interpretation accuracy based on artificial intelligence (350) by transmitting voice to the artificial intelligence (350) for real-time STT conversion to receive the STT result, and determine whether the sentence ends based on the received STT result to quickly detect the sentence (punctuation separation). At this time, when a sentence is detected, the function providing system (320) can improve the accuracy of voice interpretation by performing translation and speech synthesis on a sentence-by-sentence basis through the artificial intelligence (350).

[0061] Additionally, the function providing system (320) can correct the results of STT conversion using a large language model. For example, the function providing system (320) can improve the STT recognition rate by inputting the results of STT conversion and meeting materials and / or speech glossaries into the large language model and requesting correction of the results of STT conversion. The meeting materials can improve the STT recognition rate by providing expressions that are likely to be included in the speech. Additionally, the speech glossary can provide individual speech characteristics to the large language model using a dataset (pairs of text corresponding to speech) collected along with individual speech utterances, thereby increasing the individual STT recognition rate. The large language model can also be included in the artificial intelligence (350) described above.

[0062] Additionally, the function providing system (320) can detect materials directly or indirectly requested by participants from voice input in real time and from meeting content being recorded in real time, and can provide the detected materials in real time by expressing them in various media through artificial intelligence (350). For example, directly requested materials may be provided by voice in which a participant directly requests that materials be provided, and indirectly requested materials may be provided as an answer to a participant's question in cases where no one provides an answer voice to a specific participant's question, even though the participant did not directly request the provision of materials. Such answers may be generated by artificial intelligence (350) based on the participant's question and the meeting content being recorded in real time, and / or meeting materials, etc. Various media may include documents in Markdown format, documents in LaTeX format, executable code, and / or images.

[0063] Additionally, the function providing system (320) can recognize human emotions through facial expressions in images transmitted and received via video conferencing, and further correct the STT conversion result and / or translation result based on the recognized emotion. For example, the function providing system (320) can recognize a user's smiling facial expression to determine whether the user's voice is a joke and correct the STT conversion result and / or text translation result for the voice.

[0064] Additionally, the function providing system (320) may include a function to recognize text of a document appearing on a shared screen during a video conference using Optical Character Recognition (OCR) technology and to provide the recognized text or a real-time translation result of the text. At this time, the real-time translation of the content recognized using OCR technology may also be performed using artificial intelligence (350).

[0065] FIGS. 8 and 9 are drawings illustrating an example of implementation of a function providing system in an embodiment of the present invention. An open source container orchestrator (810) may be implemented in the form of Linux servers. These Linux servers may include the previously described web API server (710) and front-end server (811). A virtual machine (820) may include a plurality of virtual machines. One virtual machine (e.g., virtual machine 1 (821)) may include Docker (822), and Docker (822) may include a plurality of containers. One container (e.g., container (823)) may include an audio delivery (824), and each audio delivery (824) may correspond to a bot (e.g., an interpreter bot) that receives and processes audio according to a configuration. Additionally, one virtual machine may further include an audio delivery manager (825).

[0066] The web API server (710) is a server that manages the entire system and can process CRUD (Create, Read, Update, Delete) operations for meeting minutes. As previously described, the web API server (710) can directly request a meeting minutes summary from the artificial intelligence (350). Additionally, the web API server (710) can process CRUD operations for meeting comments entered by participants in the video conferencing service. Furthermore, the web API server (710) can process the invitation and termination of bots. In this case, the web API server (710) can post the relevant events to the audio delivery manager (825) and receive the current status of the audio delivery manager (825).

[0067] The front-end server (811) allows a user to log in through authentication of the video conferencing service provided by the video conferencing system (310), and the login session may include an API token for the REST (Representational State Transfer) API of the video conferencing service. The user may be identified through the account information of the video conferencing service.

[0068] The virtual machine (820) can be implemented to utilize a Windows-based server. The audio delivery (824) can be executed in a container (823) containing a Windows-based Docker (822). The audio delivery (824) can receive audio (e.g., user voice) from the video conferencing system (310) in an audio callback using the SDK of the video conferencing service, and can deliver audio (e.g., interpreted synthetic voice) to the video conferencing system (310) using the SDK API.

[0069] The audio delivery manager (825) can run on Windows and can execute Docker commands requested by the web API server (710). At startup, the audio delivery manager (825) can start long-term polling to the web API server (710). This reduces the time required to allocate audio delivery because multiple containers are managed through a container pool, and the audio delivery manager (825) can allocate and operate audio delivery (interpreter bots) managed in the container pool in real time. When an event is published from the web API server (710), the audio delivery manager (825) can execute Docker commands. The audio delivery manager (825) can create and delete Docker containers (e.g., container (823)) based on a Windows Docker image. The audio delivery manager (825) can periodically send the status of a Windows-based virtual machine (e.g., virtual machine 1 (821)) to the web API server (710).

[0070] The database (730) is a general-purpose database and may contain meeting data such as minutes, comments, and cached summaries. All access to the database (730) can be made through the web API server (710).

[0071] Redis (830) is a shared cache and can be used for bidirectional communication between the audio delivery (824) and the web API server (710) as shown in FIG. 9. For example, Redis (830) can be used for bidirectional communication between the audio delivery (824) and the web API server (710) for various settings (speech rate, VAD (Voice Activity Detection) model parameters, etc.) when a video conference is executed.

[0072] OpenSearch (840) can be used to store logs. Audio Delivery (824) can send logs directly to Audio Delivery (824) via HTTP. Audio Delivery Manager (825) can send logs to OpenSearch (840) via the 'go-opensearch' package. Web API Server (710) and Front End Server (811) can send logs to OpenSearch (840) via a separately implemented log transmitter.

[0073] FIG. 10 is a flowchart illustrating an example of a method for providing a video conferencing service in an embodiment of the present invention. The method for providing a video conferencing service according to the present embodiment may be performed by a video conferencing service providing system (300) implemented through at least one computer device. At this time, each of the at least one computer device may correspond to the computer device (200) described above through FIG. 2. At least one processor included in the at least one computer device may be implemented to execute a control instruction according to the code of an operating system included in memory or the code of at least one computer program. Here, the at least one processor may operate according to the control instruction provided by the code to control the video conferencing service providing system (300) so that the video conferencing service providing system (300) performs steps (1010 to 1050) included in the method of FIG. 10.

[0074] In step (1010), the video conferencing service providing system (300) may set up two or more interpreter bots participating as virtual participants in the video conferencing service. For example, the video conferencing service providing system (300) may set up at least one of the output language of each of the two or more interpreter bots and at least one of the voice of each of the two or more interpreter bots for each participant using at least one of the participants' names, the participants' voices, the language used by the participants, and the participants' countries. Here, setting up the output language of each interpreter bot may be determining which language interpreter bots will be used. Additionally, setting up the voice of each participant may be determining which voice the interpreted voice will be output when a specific interpreter bot outputs an interpreted voice for a participant's voice.

[0075] In step (1020), the video conferencing service providing system (300) can interpret the voice of a first language input by participants of the video conferencing service into the voice of at least one other language different from the first language through the process of voice recognition, translation, and synthetic voice generation between two or more interpreter bots and artificial intelligence. For example, the video conferencing service providing system (300) can generate text of the first language by recognizing the voice of the first language through artificial intelligence and a first interpreter bot implemented to process the voice of the first language among two or more interpreter bots. As a more specific example, the video conferencing service providing system (300) can generate text of the first language by requesting STT conversion for the voice of the first language to artificial intelligence through the first interpreter bot, thereby recognizing the voice of the first language through artificial intelligence. Additionally, the video conferencing service providing system (300) can translate the text of the first language into text of the second language through artificial intelligence and a second interpreter bot implemented to process the voice of the second language among two or more interpreter bots. For example, the video conferencing service providing system (300) can translate text of the first language into text of the second language through artificial intelligence by requesting translation of text of the first language into text of the second language through the second interpretation bot. Additionally, the video conferencing service providing system (300) can convert text of the second language into speech of the second language through the second interpretation bot and artificial intelligence. For example, the video conferencing service providing system (300) can convert text of the second language into speech of the second language through artificial intelligence by requesting TTS conversion of text of the second language through the second interpretation bot.

[0076] In the case where there are three languages ​​used in the video conferencing service, the video conferencing service providing system (300) can translate text of the first language into text of the third language through a third interpreter bot implemented to process the voice of the third language among two or more interpreter bots and artificial intelligence, and can convert text of the third language into voice of the third language through the third interpreter bot and artificial intelligence.

[0077] Meanwhile, as previously explained, speech recognition can be achieved through STT conversion using artificial intelligence. In this case, the video conferencing service providing system (300) can transmit the speech of the first language to the artificial intelligence in real time for STT conversion in step (1020) to receive the STT result, and determine whether the sentence ends based on the received STT result. Through this, the video conferencing service providing system (300) can improve the interpretation accuracy of the artificial intelligence by performing translation and speech synthesis using artificial intelligence on the text in sentence units detected based on whether the sentence ends.

[0078] Additionally, the video conferencing service providing system (300) can correct the results of speech recognition using artificial intelligence by inputting the results of speech recognition into a large language model along with at least one of the meeting materials and the participants' speech glossary. In this case, the process of translation and speech synthesis can be processed based on the corrected results of speech recognition. Here, the speech glossary may include pairs of the participants' speech utterances and texts corresponding to the speech utterances.

[0079] Additionally, the video conferencing service providing system (300) can recognize emotions through facial expressions of people in images transmitted and received through the video conferencing service, and can correct at least one of the results of speech recognition and translation based on the recognized emotions.

[0080] In step (1030), the video conferencing service providing system (300) may provide an interpreted voice. For example, the video conferencing service providing system (300) may provide an interpreted voice so that the interpreted voice is output through a channel corresponding to the language. For example, the video conferencing service providing system (300) may provide a voice of the second language so that the voice of the second language is output through a channel for the second language, and may provide a voice of the third language so that the voice of the third language is output through a channel for the third language. At this time, the voice of the second language, which is an interpretation of the voice of the first language, may be output to a participant who has selected a channel for the second language, and the voice of the third language, which is an interpretation of the voice of the first language, may be output to a participant who has selected a channel for the third language.

[0081] In step (1040), the video conferencing service providing system (300) can provide directly requested materials or indirectly requested materials in real time based on meeting content recorded from voice input in real time through the video conferencing service. At this time, the directly requested materials include materials requested by the voice input through the video conferencing service, and the indirectly requested materials may include answers generated based on artificial intelligence for questions that other participants have not answered. At this time, the video conferencing service providing system (300) can provide the materials in at least one form among a document in Markdown format, a document in LaTeX format, executable code, and an image.

[0082] In step (1050), the video conferencing service providing system (300) can recognize the text of a document appearing on a screen shared with participants through the video conferencing service using optical character recognition technology, and can provide the recognized text or a real-time translation result of the recognized text into another language. Through this, the video conferencing service providing system (300) can provide not only a voice interpretation function but also a real-time translation result for a document shared during the video conferencing service.

[0083] As such, according to embodiments of the present invention, a method and system for providing a video conferencing service including an artificial intelligence-based voice interpretation function can be provided.

[0084] The system or device described above may be implemented as a hardware component, or a combination of a hardware component and a software component. For example, the device and component described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0085] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be embodied in any type of machine, component, physical device, virtual equipment, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0086] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either individually or in combination. The medium may continuously store a program executable by a computer, or temporarily store it for execution or download. Furthermore, the medium may be various recording or storage means in the form of a single or multiple hardware components, and is not limited to a medium directly connected to a computer system, but may exist distributed over a network. Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and media configured to store program instructions, including ROM, RAM, and flash memory. Additionally, other examples of media may include recording or storage media managed by app stores that distribute applications or sites and servers that supply or distribute various other software. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0087] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0088] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

1. A method for providing a video conferencing service of a video conferencing service providing system implemented by at least one computer device, The above at least one computer device includes at least one processor, and The above method of providing a video conferencing service is, A step of setting up two or more interpreter bots participating as virtual participants in a video conferencing service by the above-mentioned at least one processor; A step of interpreting a voice of a first language input by participants of the video conferencing service into a voice of at least one other language different from the first language through the process of speech recognition, translation, and synthetic speech generation between the two or more interpreter bots and artificial intelligence by the at least one processor; and The step of providing the interpreted voice by the above at least one processor A method for providing a video conferencing service characterized by including 2. In Paragraph 1, The step of setting up two or more of the above-mentioned interpreter bots is, A step of setting at least one of the output language of each of the two or more interpretation bots and the voice of each participant of each of the two or more interpretation bots using at least one of the names of the participants, the voices of the participants, the language used by the participants, and the countries of the participants. A method for providing a video conferencing service characterized by including 3. In Paragraph 1, The above interpretation step is, A step of recognizing the speech of the first language and generating text in the first language through the artificial intelligence and a first interpreter bot implemented to process the speech of the first language among the two or more interpreter bots above; A step of translating text of the first language into text of the second language through the artificial intelligence and a second interpreter bot implemented to process the voice of the second language among the two or more interpreter bots; and A step of converting text of the second language into speech of the second language through the second interpretation bot and the artificial intelligence. A method for providing a video conferencing service characterized by including 4. In Paragraph 3, The above interpretation step is, A step of translating text in a first language into text in a third language through a third interpretation bot implemented to process speech of a third language among the two or more interpretation bots above and the artificial intelligence above; and A step of converting text in a third language into speech in a third language through the aforementioned third interpretation bot and the aforementioned artificial intelligence. A method for providing a video conferencing service characterized by further including 5. In Paragraph 4, The step of providing the above-mentioned interpreted voice is, Provides a second language voice so that the second language voice is output through a channel for the second language, and provides a third language voice so that the third language voice is output through a channel for the third language, The audio of the second language, interpreted from the audio of the first language, is output to the participant who selected the channel for the second language, and the audio of the third language, interpreted from the audio of the first language, is output to the participant who selected the channel for the third language. A method for providing a video conferencing service characterized by 6. In Paragraph 3, The step of generating the above text is, Requesting STT (Speech-To-Text) conversion of the speech of the first language to the artificial intelligence through the above-mentioned first interpretation bot, recognizing the speech of the first language through the artificial intelligence, and generating text of the first language, The above translation step is, A request for translation of the text in the first language into the text in the second language is made to the artificial intelligence through the above second interpretation bot, and the text in the first language is translated into the text in the second language through the said artificial intelligence, and The above conversion step is, Requesting Text-to-Speech (TTS) conversion of text in the second language to the artificial intelligence through the second interpretation bot, and converting the text in the second language into speech in the second language through the artificial intelligence. A method for providing a video conferencing service characterized by 7. In Paragraph 1, The above speech recognition is performed through STT (Speech-To-Text) conversion using the above artificial intelligence, and The above interpretation step is, A step of transmitting the speech of the first language to the artificial intelligence in real time for the STT conversion to receive the STT result, and determining whether to end the sentence according to the received STT result; and A step of performing the translation and speech synthesis using the artificial intelligence on sentence-unit text detected according to whether the above sentence ends. A method for providing a video conferencing service characterized by including 8. In Paragraph 1, The above interpretation step is, A step of inputting the result of the speech recognition using the artificial intelligence into a Large Language Model (LLM) along with at least one of meeting materials and a participant's speech glossary to correct the result of the speech recognition. A method for providing a video conferencing service characterized by including 9. In Paragraph 8, A method for providing a video conferencing service characterized by the above-mentioned speech glossary including pairs of a participant's speech utterance and text corresponding to the speech utterance.

10. In Paragraph 1, The above method of providing a video conferencing service is, A step of providing directly requested data or indirectly requested data in real time based on meeting content recorded from voice being input in real time through the video conferencing service by the above-mentioned at least one processor. A method for providing a video conferencing service characterized by further including 11. In Paragraph 10, The directly requested data mentioned above includes the data requested by the voice input through the video conferencing service, and The above indirectly requested data includes answers generated based on the above artificial intelligence for questions for which no other participants have answered. A method for providing a video conferencing service characterized by 12. In Paragraph 10, The step of providing the above data in real time is, A method for providing a video conferencing service characterized by providing the above data in at least one of a Markdown document, a LaTeX document, executable code, and an image.

13. In Paragraph 1, The above interpretation step is, A step of recognizing emotions through human facial expressions in images transmitted and received through the above-mentioned video conferencing service; and A step of correcting at least one of the result of the voice recognition and the result of the translation based on the above-mentioned recognized emotion A method for providing a video conferencing service characterized by including 14. In Paragraph 1, The above method of providing a video conferencing service is, A step of recognizing text of a document appearing on a screen shared with participants through the video conferencing service by at least one processor using Optical Character Recognition (OCR) technology; and A step of providing a real-time translation result of the recognized text or the recognized text into another language by the above at least one processor A method for providing a video conferencing service characterized by further including 15. A computer program stored on a computer-readable recording medium to execute the method of any one of claims 1 to 14 on a computer device in combination with a computer device.

16. A computer-readable recording medium having a computer program recorded thereon for executing the method of any one of paragraphs 1 through 14 on a computer device.

17. A video conferencing service providing system implemented by at least one computer device, The above at least one computer device includes at least one processor, and By the above at least one processor, Set up two or more interpretation bots to participate as virtual participants in a video conferencing service, and The voice of a first language input by participants of the above-mentioned video conferencing service is interpreted into the voice of at least one other language different from the first language through the process of speech recognition, translation, and synthetic speech generation between the two or more interpretation bots and artificial intelligence, and Providing the above-mentioned interpreted audio A video conferencing service provision system characterized by 18. In Paragraph 17, To set up the above two or more interpretation bots, by the above at least one processor, Setting at least one of the output language of each of the two or more interpretation bots and the voice of each participant of each of the two or more interpretation bots using at least one of the names of the participants, the voices of the participants, the languages ​​used by the participants, and the countries of the participants. A video conferencing service provision system characterized by 19. In Paragraph 17, In order to interpret the speech of a first language into the speech of at least one other language different from the first language, by the at least one processor, A first interpreter bot implemented to process the speech of a first language among the two or more interpreter bots above, and the artificial intelligence above recognize the speech of the first language to generate text of the first language, and Translating text of the first language into text of the second language through the second interpreter bot implemented to process the voice of the second language among the two or more interpreter bots above and the artificial intelligence above, and Converting text of the second language into speech of the second language through the second interpretation bot and the artificial intelligence. A video conferencing service provision system characterized by 20. In Paragraph 17, The above video conferencing service providing system includes a plurality of containers managed through a container pool in a Windows-based virtual machine and an audio delivery manager that manages the plurality of containers. Each of the above plurality of containers includes audio delivery for an interpreter bot, and The above audio delivery manager allocates audio delivery for an interpreter bot participating in the video conferencing service in real time by selecting a container containing the audio delivery from the container pool. A video conferencing service provision system characterized by

Images