Electronic device and method for sound object separation

Abstract

A method for operating an electronic device, according to one embodiment of the present disclosure, may comprise the operations of: preprocessing input audio data in order to acquire input data of an object separation model; inputting the input data to the object separation model so as to acquire mask data for a plurality of objects; and generating object audio data for the plurality of objects by using the mask data for the plurality of objects. The pre-processing operation can include the operations of: converting the input audio data into a frequency domain so as to acquire first frequency domain data having a first number of frequency bins; and using a designated conversion algorithm to convert the first frequency domain data into second frequency domain data having a second number of frequency bins that is less than the first number of frequency bins. The second frequency domain data can be input to the object separation model as input data.

Description

Electronic device and method for sound object separation

[0001] The present disclosure relates to an electronic device and method for sound object separation.

[0002] With the advancement of technology (e.g., AI technology), techniques for separating human voices from background sounds or separating instrument sounds from music are being proposed. These sound separation technologies enable new sound rendering methods.

[0003] However, most sound separation technologies rely on server-based models; while these models offer excellent performance, they are difficult to implement in on-device environments due to constraints such as model size and input data size. Furthermore, to achieve high performance, sound separation models primarily employ non-causal structures that utilize future data, resulting in a lack of real-time capability. Therefore, there is a need for a sound separation technology that guarantees real-time performance and can be implemented on devices.

[0004] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0005] According to one embodiment of the present disclosure, an electronic device may include a memory comprising at least one storage medium for storing instructions; and at least one processor comprising a processing circuit. The at least one processor may be configured individually and / or collectively to cause the electronic device to: preprocess input audio data to obtain input data for an object separation model; input the input data to the object separation model to obtain mask data for a plurality of objects; and generate object audio data for a plurality of objects using the mask data for the plurality of objects. The operation of preprocessing the input audio data may include: converting the input audio data into a frequency domain to obtain first frequency domain data having a first number of frequency bins; and converting the first frequency domain data into second frequency domain data having a second number of frequency bins smaller than the first number of frequency bins using a specified conversion algorithm, wherein the second frequency domain data may be input to the object separation model as input data.

[0006] A method for operating an electronic device according to one embodiment of the present disclosure may include: an operation of preprocessing input audio data to obtain input data for an object separation model; an operation of inputting the input data into the object separation model to obtain mask data for a plurality of objects; and an operation of generating object audio data for a plurality of objects using the mask data for the plurality of objects. The operation of preprocessing the input audio data may include: an operation of converting the input audio data into a frequency domain to obtain first frequency domain data having a first number of frequency bins; and an operation of converting the first frequency domain data into second frequency domain data having a second number of frequency bins smaller than the first number of frequency bins using a specified conversion algorithm, wherein the second frequency domain data may be input to the object separation model as input data.

[0007] According to one embodiment, the operation of converting to the second frequency domain data includes: applying a second frequency resolution higher than the first frequency resolution to frequency bins having a frequency greater than or equal to the reference frequency among the frequency bins of the first frequency domain data, wherein the first frequency resolution may be a frequency resolution applied to frequency bins having a frequency less than the reference frequency among the frequency bins of the first frequency domain data.

[0008] According to one embodiment, the operation of converting to the second frequency domain data includes applying a specified transformation function to frequency bins having a frequency greater than or equal to a reference frequency among the frequency bins of the first frequency domain data, wherein the transformation function is set based on the reference frequency, the number of first frequency bins, the number of second frequency bins, and the first frequency resolution, and the first frequency resolution may be set as a value obtained by dividing the sampling rate of the input audio data by the size of the FFT (fast Fourier transform).

[0009] According to one embodiment, the reference frequency may be set based on the frequency band of a first object corresponding to voice among the plurality of objects.

[0010] According to one embodiment, an FFT is used for the frequency domain transformation, the first frequency bin is half the size of the FFT, and the second frequency bin may be half the number of the first frequency bin.

[0011] According to one embodiment, the object separation model corresponds to a U-Net model using a skip connection structure, and the skip connection structure may include a structure in which the output value of each stage of the encoder is connected to the corresponding stage of the decoder.

[0012] According to one embodiment, the object separation model may have a single input / output structure.

[0013] According to one embodiment, the object separation model may include at least one 1D convolution layer, at least one 1D-Tr (transpose) convolution layer and an LSTM (long short-term memory) layer.

[0014] According to one embodiment, the object separation model is an on-device model stored in the memory, and the time duration of the input audio data can be set to a value smaller than 50ms.

[0015] According to one embodiment, the operation of generating object audio data for the plurality of objects may include: the operation of generating each mask data having the second frequency bin number using a specified restoration algorithm to obtain each restored mask data having the first frequency bin number; and the operation of generating object audio data for each of the plurality of objects by multiplying the first frequency domain data by each of the restored mask data. The plurality of objects may include at least one of a first object corresponding to voice, a second object corresponding to music, or a third object corresponding to effect.

[0016] According to one embodiment, an electronic device can ensure real-time performance by utilizing an object separation model having a single input / output structure and / or a causal structure. According to one embodiment, the electronic device can be implemented as an on-device model by reducing the model size of the object separation model and / or the size of input data in the time and frequency axes.

[0017] However, the problems to be solved in this disclosure are not limited to those mentioned above, and may be determined in various ways without departing from the spirit and scope of this disclosure.

[0018] The above and other aspects, features, and advantages of specific embodiments of the present disclosure will become more apparent from the following detailed description when considered together with the accompanying drawings:

[0019] FIG. 1 is a block diagram of an electronic device in a network environment according to one embodiment disclosed in this document.

[0020] FIG. 2 is a drawing illustrating an exemplary configuration of an electronic device according to one embodiment of the present disclosure.

[0021] FIG. 3 is a flowchart illustrating an exemplary object separation method according to one embodiment of the present disclosure.

[0022] Figure 4 is a flowchart illustrating the operation of generating exemplary model input data used for the object separation method of Figure 3.

[0023] Figure 5 is a graph illustrating a transformation function for generating the model input data of Figure 4.

[0024] FIG. 6 is a drawing illustrating an exemplary object separation model according to one embodiment of the present disclosure.

[0025] FIG. 7 is a drawing illustrating the internal configuration of an exemplary object separation model according to one embodiment of the present disclosure.

[0026] FIG. 8 is a drawing illustrating the internal configuration of an exemplary object separation model according to one embodiment of the present disclosure.

[0027] FIG. 9 is a drawing illustrating an exemplary internal configuration of an object separation model according to one embodiment of the present disclosure.

[0028] FIG. 10 is a drawing illustrating an exemplary method of using separated object audio data in an electronic device according to one embodiment of the present disclosure.

[0029] In the following description, reference is made to the attached drawings, and various embodiments that may be implemented are illustrated within the drawings. Additionally, other embodiments may be used and structural modifications may be made without departing from the scope of the present disclosure.

[0030] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, a home appliance, and the like. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0031] The embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of various embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise.

[0032] In this document, each of the phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B or C,” “at least one of A, B and C,” and “at least one of A, B, or C” may include any one of the items listed together with the corresponding phrase, or all possible combinations thereof. Terms such as “first,” “second,” or “first” or “second” may be used simply to distinguish a component from another component and do not limit the components in any other aspect (e.g., importance or order). Where any (e.g., first) component is referred to as “coupled” or “connected” to another (e.g., second) component, with or without the terms “functionally” or “communicationly,” it means that said component may be connected to said other component directly (e.g., wired), wirelessly, or through a third component.

[0033] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software or firmware, or any combination thereof, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0034] According to one embodiment, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to one embodiment, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the components of the multiple components in the same or similar manner as those performed by the corresponding components among the multiple components prior to integration. According to one embodiment, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0035] FIG. 1 is a block diagram of an exemplary electronic device in a network environment according to one embodiment disclosed in this document.

[0036] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In one embodiment, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In one embodiment, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0037] The processor (120) can control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., a program (140)), and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof. Accordingly, each processor (120) or “model” in this disclosure may include a processing circuit and / or a plurality of processors. For example, as used in this disclosure, including in the claims, the term “processor” or “model” may include various processing circuits including at least one processor, wherein at least one of the at least one processor may be configured to perform the various functions described in this disclosure individually and / or collectively in a distributed manner.In the present disclosure, where "one processor," "at least one processor," "one model," "at least one model," and "one or more processors" are described as being configured to perform a plurality of functions, these terms include, for example, without limitation, a situation in which one processor and / or model performs some of the described functions and other processor(s) and / or model(s) perform other parts of the described functions, and a situation in which a single processor and / or model can perform all of the described functions. Additionally, at least one processor may include a combination of processors performing various functions among the described or disclosed functions, for example, in a distributed manner. At least one processor may execute program instructions to achieve or perform various functions. Likewise, at least one model may include a combination of circuits and / or processors performing various functions among the described or disclosed functions, for example, in a distributed manner. At least one processor and / or model may execute program instructions to achieve or perform various functions.

[0038] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence is performed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0039] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0040] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0041] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0042] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0043] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a hall area-gram device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0044] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0045] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0046] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0047] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0048] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that can be perceived by the user through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0049] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0050] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0051] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0052] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0053] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) may support a Peak data rate (e.g., 20 Gbps or more) for eMBB realization, loss coverage (e.g., 164 dB or less) for mMTC realization, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for URLLC realization.

[0054] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to one embodiment, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197). According to one embodiment, the antenna module (197) may form a mmWave antenna module. According to one embodiment, a mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0055] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0056] According to one embodiment, commands or data may be transmitted or received between an electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0057] FIG. 2 is a drawing illustrating an exemplary configuration of an electronic device according to one embodiment of the present disclosure.

[0058] According to one embodiment, an electronic device (200) (e.g., the electronic device (101) of FIG. 1) can obtain object audio data (241, 242, 243) for a plurality of objects from input sound data (e.g., input audio data (201)) using an object separation model (221). In this disclosure, for convenience of explanation, the input sound data is described as an example of input audio data (201), but is not limited thereto. For example, various types of sound data may be used for object separation.

[0059] According to one embodiment, the object separation model (221) may include various circuits and / or executable program instructions, and may include, for example, a model used to obtain mask data for a plurality of objects from input audio data (201). The object separation model (221) may be, for example, an artificial intelligence (AI) model trained to receive input data in the frequency domain generated based on the input audio data (201) and output mask data for a plurality of objects. The object separation model (221) may be, for example, a frequency domain convolution-based AI model (e.g., a neural network model).

[0060] According to one embodiment, the mask data may be data acting as a filter used to separate object audio data for a plurality of objects from input audio data, for example. The mask data may be in the form of a tensor (or matrix) having values ​​between 0 and 1, for example.

[0061] Referring to FIG. 2, the electronic device (200) may include an audio input pre-processor (210), an object mask extractor (220), and / or an object audio generator (230). According to one embodiment, the electronic device (200) may be a multimedia device or display device that processes and / or outputs a sound signal. For example, the electronic device (200) may be a multimedia device such as a TV, smartphone, tablet, laptop, desktop, monitor, soundbar, Bluetooth speaker, home theater system, projector, electronic whiteboard, streaming device, but is not limited thereto.

[0062] According to one embodiment, an audio input preprocessor (210) can preprocess input audio data (201) (or signal) to obtain (or generate) input data for an object mask extractor (220) (or object separation model (221)). In the present disclosure, the input audio data may be referred to as original audio data.

[0063] According to one embodiment, the audio input preprocessor (210) may be implemented by at least one processor (e.g., the processor (120) of FIG. 1). For example, each component of the audio input preprocessor (210) may be implemented by a digital signal processor (DSP).

[0064] According to one embodiment, the audio input preprocessor (210) may include a frequency domain converter (211), a normalizer (212), and / or an input converter (213), each of which may include various circuits and / or executable program instructions.

[0065] According to one embodiment, the frequency domain converter (211) can convert input audio data (201) into the frequency domain to obtain first frequency domain data having a first number of frequency bins. For example, the frequency domain converter (211) can convert the input audio data (201) into the frequency domain by applying an FFT to the input audio data. According to one embodiment, the input audio data (201) may be audio data having a sampling rate of 48 kHz. According to one embodiment, the input audio data (201) may be audio data set to a specified time length (e.g., 50 ms) or less. When the input audio data (201) having such a limited time length is used for object separation, the time axis size (length) of the data input to the object separation model (221) may be reduced. Through this, the size of the object separation model (221) may be reduced.

[0066] According to one embodiment, the number of first frequency bins may be half the size of the FFT. For example, if the FFT size is 2048, the number of first frequency bins may be 1024. The FFT size may be the number of FFT-points.

[0067] According to one embodiment, a normalizer (212) can perform normalization on the first frequency domain data to obtain the normalized first frequency domain data. For example, the normalizer (212) can perform normalization on the first frequency domain data using a PCEN (per-channel energy normalization) technique. Through this, frequency domain data robust to noise can be obtained. According to one embodiment, the first frequency domain data can be multiplied by the normalized first frequency domain data through a multiplier (x) to obtain scaled first frequency domain data. Through this, important frequency components can be emphasized and unnecessary frequency components can be suppressed. In the present disclosure, the normalized first frequency domain data and the scaled first frequency domain data may have the same first frequency bin number as the first frequency domain data. In the present disclosure, frequency domain data having the first frequency bin number may be collectively referred to as the first frequency domain data.

[0068] According to one embodiment, the input converter (213) may convert a first frequency domain data (e.g., scaled first frequency domain data) into a second frequency domain data having a second frequency bin number smaller than the first frequency bin number using a specified conversion algorithm. According to one embodiment, the second frequency bin number (e.g., 512) may be half the first frequency bin number (e.g., 1024).

[0069] According to one embodiment, the input converter (213) can obtain second frequency domain data having a second number of frequency bins by applying a second frequency resolution higher than the first frequency resolution to frequency bins having a frequency greater than or equal to the reference frequency among the frequency bins of the first frequency domain data. According to one embodiment, the first frequency resolution may be a frequency resolution applied to frequency bins having a frequency less than the reference frequency among the frequency bins of the first frequency domain data. The first frequency resolution may be set to a value obtained by dividing the sampling rate of the input audio data (e.g., 48 kHz) by the FFT size (e.g., 2048).

[0070] According to one embodiment, the input converter (213) can obtain second frequency domain data having a second number of frequency bins by applying a specified conversion function to frequency bins among the frequency bins of the first frequency domain data that have a frequency greater than or equal to a reference frequency. According to one embodiment, the conversion function may be set based on a reference frequency, a first number of frequency bins, a second number of frequency bins, and / or a first frequency resolution. As described above, the first frequency resolution may be set to a value obtained by dividing the sampling rate of the input audio data (e.g., 48 kHz) by the FFT size (e.g., 2048). An example of a specified conversion function is described in more detail below with reference to FIG. 5.

[0071] According to one embodiment, a reference frequency may be set based on the frequency band of a first object corresponding to speech among a plurality of objects extracted from input audio data. For example, the reference frequency may be set to 10 kHz (or a frequency within a first range from 10 kHz). This is because, since most of the frequency components of the first object corresponding to speech do not exceed 10 kHz, even if frequency domain data is converted by setting a high frequency resolution in a frequency band of 10 kHz or higher, it may not significantly affect the performance of object separation.

[0072] According to one embodiment, the second frequency domain data can be used as input data for an object mask extractor (220). For example, the second frequency domain data can be input into an object separation model (221) included in the object mask extractor (220). When the second frequency domain data with a reduced number of frequency bins is input into the object separation model (221), the frequency axis size (length) of the input data of the object separation model (221) can be reduced compared to when the first frequency domain data is input into the object separation model (221) as is. This allows the model size of the object separation model (221) to be reduced. In the present disclosure, the data input into the object separation model (221) may be referred to as model input data.

[0073] According to one embodiment, the object mask extractor (220) can obtain mask data for a plurality of objects using input data (e.g., second frequency domain data) transmitted from the audio input preprocessor (210). For example, the object mask extractor (220) can input the input data into an object separation model (221) and obtain mask data for a plurality of objects output from the object separation model (221). The mask data is frequency domain data and may have the same number of second frequency bins as the second frequency domain data. The plurality of objects may include, for example, at least one of a first object corresponding to speech, a second object corresponding to music, or a third object corresponding to sound effects.

[0074] According to one embodiment, the object mask extractor (220) may be implemented by at least one processor (e.g., the processor (120) of FIG. 1). For example, each component of the object mask extractor (220) may be implemented by a neural processing unit (NPU).

[0075] According to one embodiment, the object separation model (221) may be a U-Net model utilizing a skip connection structure. The skip connection structure may be a structure in which the output value of each stage of the encoder is connected to the corresponding stage of the decoder. Through this skip connection structure, the recovery performance of the decoder of the object separation model (221) may be improved.

[0076] According to one embodiment, the object separation model (221) may have a single input / output structure. By implementing an object separation model having a single input / output structure, real-time performance is guaranteed, and it may be easily implemented on a device.

[0077] According to one embodiment, the object separation model (221) may include, for example, and without limitation, at least one 1D convolution layer, at least one 1D-Tr (transpose) convolution layer and / or LSTM layer. The at least one 1D convolution layer may not convolve the time axis, but may convolve and / or deconvolve only the frequency axis. The at least one 1D convolution layer may be used to compress and decompress only the frequency axis information. The LSTM layer may store time axis information and learn the influence of a single input data of a short time (e.g., within 50ms) input in real time and the influence of previously input past input data. Through this LSTM layer, the time axis size (length) of the input data can be reduced. In this way, by implementing the object separation model (221) with a 1D convolution layer, a 1D-Tr convolution layer, and an LSTM layer, a model that has a single input / output structure and can be implemented on a device can be implemented.

[0078] According to one embodiment, the object separation model (221) may be an on-device model stored in the memory of an electronic device (e.g., the memory (130) of FIG. 1), but is not limited thereto. For example, if necessary, the object separation model (221) may be a model stored in a server (e.g., the server (108) of FIG. 1).

[0079] An object separation model (221) having the above-described characteristic(s) can be easily implemented on a device and ensures real-time performance compared to a model that uses long historical input data (e.g., input data of 3 seconds or more) and includes 2D convolution layers and transformers (e.g., Demucs (deep extractor for music sources) model). An example of such an object separation model (221) will be described in more detail below with reference to FIGS. 6 through 9.

[0080] According to one embodiment, the object audio generator (230) can generate object audio data for a plurality of objects using mask data for a plurality of objects transmitted from the object mask extractor (220).

[0081] According to one embodiment, the object audio generator (230) may be implemented by at least one processor (e.g., the processor (120) of FIG. 1, the description thereof applies here as well as above). For example, each component of the object audio generator (230) may be implemented by a DSP.

[0082] According to one embodiment, the object audio generator (230) may include an input recoverer (231) and / or a time domain converter (232), each of which may include various circuits and / or executable program instructions.

[0083] According to one embodiment, the input restorer (231) can restore each mask data having a second frequency bin count (e.g., 512) using a specified restoration algorithm to obtain each restored mask data having a first frequency bin count (e.g., 1024) which is greater than the second frequency bin count. The specified restoration algorithm may be, for example, an algorithm for performing the inverse operation of a specified transformation algorithm. Through this, the frequency axis size (length) reduced for input to the object separation model (221) can be restored to the original frequency axis size (length).

[0084] According to one embodiment, each restored mask data can be multiplied by the first frequency domain data through a multiplier (x) to generate frequency domain object audio data for a plurality of objects.

[0085] According to one embodiment, the time domain converter (232) can obtain each object audio data in the time domain by converting each object audio data in the frequency domain into the time domain. For example, the time domain converter (232) can convert each object audio data in the frequency domain into the time domain by applying an inverse FFT (IFFT) to each object audio data in the frequency domain. Each object audio data in the time domain may have a time length equal to the time length of the input audio data (e.g., 42.6 ms). The operation of the time domain converter (232) (e.g., the operation of applying the IFFT of the time domain converter (232)) may be the inverse operation of the operation of the frequency domain converter (211) (e.g., the operation of applying the FFT of the frequency domain converter (211)). Through this, the original input audio data (201) can be completely separated into object audio data (241, 242, 243) of each object. For example, input audio data (201) can be separated into object audio data (241) corresponding to a voice object, object audio data (242) corresponding to a music object, and object audio data (243) corresponding to a sound effect object. In this way, single object audio signal(s) for a single input audio signal can be separated in real time.

[0086] FIG. 3 is a flowchart illustrating an exemplary object separation method according to one embodiment of the present disclosure.

[0087] Figure 4 is a flowchart illustrating an exemplary operation for generating model input data used for the object separation method of Figure 3.

[0088] Figure 5 is a graph illustrating a transformation function for generating the model input data of Figure 4.

[0089] Referring to FIGS. 3 through 5, in operation 310, an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (200) of FIG. 2) may preprocess input audio data to obtain input data of an object separation model (e.g., object separation model (221) of FIG. 2). According to one embodiment, operation 310 is performed by an audio input preprocessor (e.g., audio input preprocessor (210) of FIG. 2) and may include all or part of the above-described operations performed by each component of the audio input preprocessor.

[0090] According to one embodiment, the operation of preprocessing input audio data of operation 310 may include an operation (410) of converting input audio data into the frequency domain to obtain first frequency domain data having a first frequency bin count, and / or an operation (420) of converting the first frequency domain data into second frequency domain data having a second frequency bin count smaller than the first frequency bin count using a specified conversion algorithm. According to one embodiment, an FFT is used for frequency domain conversion, and the first frequency bin count (e.g., 1024) may be half the FFT size (e.g., 2048) of the FFT, and the second frequency bin count (e.g., 512) may be half the first frequency bin count. The second frequency domain data with a reduced frequency axis length may be input to an object separation model as input data (model input data). Through this, the size of the input data of the object separation model and the size of the model may be reduced.

[0091] According to one embodiment, operation 410 may be performed by a frequency domain converter (e.g., the frequency domain converter (211) of FIG. 2). According to one embodiment, operation 420 may be performed by an input converter (e.g., the input converter (213) of FIG. 2).

[0092] According to one embodiment, a second frequency domain data having a second number of frequency bins can be obtained by applying a second frequency resolution higher than the first frequency resolution to frequency bins having a frequency greater than or equal to the reference frequency among the frequency bins of the first frequency domain data. The first frequency resolution may be a frequency resolution applied to frequency bins having a frequency less than the reference frequency among the frequency bins of the first frequency domain data. The first frequency resolution may be set as a value obtained by dividing the sampling rate of the input audio data by the FFT size (e.g., 23.4Hz = 48kHz / 2048). According to one embodiment, the reference frequency may be set based on the frequency band of a first object corresponding to speech among a plurality of objects.

[0093] According to one embodiment, an electronic device can obtain second frequency domain data having a second number of frequency bins by applying a specified transformation function to frequency bins having a frequency greater than or equal to a reference frequency among the frequency bins of first frequency domain data. According to one embodiment, the transformation function may be set based on a reference frequency, a first number of frequency bins, and a first frequency resolution.

[0094] Hereinafter, with reference to FIG. 5, an example of a conversion function and a conversion operation using the conversion function will be described. For convenience of explanation, it is assumed that the first frequency bin is 1024, the second frequency bin is 512 (half of the first frequency bin), and the reference frequency is 10 kHz. In this case, the conversion function can be applied from the frequency bin with index 427, which corresponds to the frequency (or frequency band) of 10 kHz, to the frequency bin with index 1024.

[0095] According to one embodiment, as illustrated in FIG. 5, the transformation function may have the form of an exponential function. For example, the transformation function may be as shown in Equation 1 below.

[0096] [Mathematical Formula 1]

[0097]

[0098] Here,

[0099] x is the index value of the frequency bin (converted bin) of the second frequency domain data.

[0100] y is the index value of the frequency bin (original bin) of the first frequency domain data corresponding to the x value, where the converted bin corresponding to the original bin has the same frequency (or frequency band).

[0101] a is the base value of the exponential function

[0102] k is the translation value

[0103] r is the frequency resolution, where frequency resolution is the value obtained by dividing the sampling rate of the input audio data by the FFT size (e.g., 23.4Hz = 48000Hz / 2048).

[0104] According to one embodiment, the values ​​of a and k may be set based on a reference frequency, a first frequency bin number, a second frequency bin number, and / or frequency resolution. For example, among the frequency bins of the first frequency domain data, a conversion function is applied from a frequency bin with an index of 427 corresponding to a reference frequency of 10 kHz to a frequency bin with an index of 1024 corresponding to 24 kHz, so that the frequency bin with index 427 is converted to a frequency bin with index 512, respectively, and the value of a may be set to 1.0104 and the value of k may be set to 463. In this case, as exemplified in the extended portion of FIG. 5, the frequency bin of index 427, the frequency bin of index 434, ..., the frequency bin of index 1015, and the frequency bin of index 1024 of the first frequency domain data can be converted into the frequency bin of index 427, the frequency bin of index 428, ..., the frequency bin of index 511, and the frequency bin of index 512 of the second frequency domain data, respectively. In this case, the frequency bin of index 427 of the second frequency domain data can correspond to a frequency (or frequency band) of 10 kHz, and the frequency bin of index 512 of the second frequency domain data can correspond to a frequency (or frequency band) of 24 kHz. Through this conversion process, the number of frequency bins is reduced, and the size of the input data input to the object separation model can be reduced.

[0105] In operation 320, the electronic device inputs the acquired input data into an object separation model (e.g., the object separation model (221) of FIG. 2) to acquire mask data for a plurality of objects. Operation 320 may include all or part of the above-described operations performed by each component of an object mask extractor (e.g., the object mask extractor (220) of FIG. 2).

[0106] According to one embodiment, the object separation model may correspond to a U-Net model utilizing a skip connection structure. The skip connection structure may be a structure in which the output value of each stage of the encoder is connected to the corresponding stage of the decoder. According to one embodiment, the object separation model may have a single input / output structure. According to one embodiment, the object separation model may include at least one 1D convolution layer, at least one 1D-Tr convolution layer, and / or a long short-term memory (LSTM) layer. According to one embodiment, the object separation model may be an on-device model stored in memory. An example of an object separation model having the above-described feature(s) is described in more detail below with reference to FIGS. 6 to 9.

[0107] In operation 330, the electronic device may generate object audio data for a plurality of objects using mask data for a plurality of objects. Operation 330 may include all or part of the above-described operations performed by each component of an object audio generator (e.g., the object audio generator (230) of FIG. 2).

[0108] According to one embodiment, an electronic device may restore each mask data having a second frequency bin number using a specified restoration algorithm to obtain each restored mask data having a first frequency bin number, and multiply each of the first frequency domain data by each of the restored mask data to generate object audio data for a plurality of objects. The plurality of objects may include at least one of a first object corresponding to speech, a second object corresponding to music, or a third object corresponding to sound effects.

[0109] FIG. 6 is a drawing illustrating an exemplary object separation model according to one embodiment of the present disclosure.

[0110] FIGS. 7 to 9 are drawings illustrating exemplary internal configurations of an object separation model according to one embodiment of the present disclosure.

[0111] Referring to FIGS. 6 through 9, according to one embodiment, the object separation model (221) may have a single input / output structure. For example, as illustrated in FIG. 6, the object separation model (221) may be a model that receives a single input (601) and outputs a single output (602). The single input (601) and the single output (602) may be set as a single frame having a specified time length in the time domain (e.g., a time length of 50ms or less). For example, as illustrated in FIG. 7, frequency domain data (601a) (e.g., the second frequency domain data of FIG. 2) for one frame (or one time step) may be input to the object separation model (221) as a single input (601), and mask data (602a, 602b, 602c) for the corresponding frame (e.g., mask data for multiple objects of FIG. 2) may be output as a single output (602). The mask data included in a single output (602) may include a first mask data (602a) corresponding to a voice object, a second mask data (602b) corresponding to a music object, and / or a third mask data (602c) corresponding to a sound effect object. The time length of one frame (or, one time step) may be the same as the time length of the input audio data (e.g., a time length of 42.6 ms).

[0112] According to one embodiment of the present disclosure, the object separation model (221) can be implemented to have a single input / output structure, thereby ensuring real-time performance. For example, the object separation model (221) having a single input / output structure can ensure real-time performance compared to the case where the object separation model has a batch input / output structure that includes multiple inputs and multiple outputs (e.g., inputs and outputs set as multiple frames with a time length of 3 seconds or more), or an input / output structure that outputs a single output using input data that includes a current single input (single frame) and past inputs (multiple frames).

[0113] According to one embodiment of the present disclosure, the object separation model (221) of the present disclosure is implemented to have a single input / output structure that uses input / output data with a short time length (e.g., the time length of one frame corresponding to 42.6 ms), thereby reducing the size (length) of the model input data and making it easy to implement on-device.

[0114] According to one embodiment, as illustrated in FIGS. 7 and 8, the object separation model (221) may include an encoder (710), an LSTM block (720), and / or a decoder (730).

[0115] According to one embodiment, an encoder (710) can obtain encoded data by using at least one 1D-CNN block (711) to extract and compress key features of input data in the frequency domain (e.g., second frequency domain data of FIG. 2). For example, as illustrated in FIG. 8, the encoder (710) can sequentially apply N 1D-CNN blocks to the input data to progressively extract and compress key features of the input data. A 1D-CNN block may include at least one 1D convolution layer. Through processing by this encoder (e.g., convolution processing on the frequency axis), frequency axis information, rather than time axis information, is compressed, and a high-level representation can be generated.

[0116] According to one embodiment, the LSTM block (720) may include at least one LSTM layer. The LSTM block (720) (or LSTM layer) may store time-axis information. The LSTM block (720) (or LSTM layer) may include LSTM cells, process input data in the time-axis direction, and learn both long-term dependencies and short-term dependencies. For example, the LSTM block (720) (or LSTM layer) may learn the influence of a single input data input in real time and the influence of previously input past input data(s). Through the implementation of an object separation model (221) using such an LSTM block (720), the time-axis size (length) of the input data may be reduced.

[0117] According to one embodiment, the decoder (730) can restore the encoded data using at least one 1D-Tr (Transpose) CNN block (731). For example, as illustrated in FIG. 8, the decoder (730) can gradually restore the encoded data by sequentially applying N 1D-TrCNN blocks to the encoded data. The 1D-TrCNN block may include at least one 1D transpose convolution layer. Through processing by this decoder (e.g., de-convolution processing on the frequency axis), the compressed encoded data can be restored.

[0118] According to one embodiment, the object separation model (221) may be a model utilizing a skip connection structure. The skip connection structure may be a structure in which the output value (output data) of each stage of the encoder is connected to the corresponding stage of the decoder. For example, as illustrated in FIGS. 7 and 8, the object separation model (221) may include a skip connection (740) in which the output value of each 1D-CNN block (711-1, 711-2, ..., 711-N; 711) of the encoder (710) is connected to each corresponding 1D-TrCNN block (731-1, 731-2, ..., 731-N; 731) of the decoder (730). In this case, each 1D-TrCNN block (731) may generate its own output data by combining its own input data with the output data of the corresponding 1D-CNN block (711). For example, as illustrated in FIG. 8, the output data of 1D-CNN block 2 (711-2) can be transmitted to 1D-TrCNN block 2 (731-2) through a skip connection (740), and 1D-TrCNN block 2 (731-2) can combine the input data of 1D-TrCNN block 2 (731-2) and the output data of 1D-CNN block 2 (711-2) to generate the output data of 1D-TrCNN block 2 (731-2), and transmit the generated output data to 1D-TrCNN block 1 (731-1). Through these skip connections (740), low-level features extracted from each stage (or block) of the encoder (710) are transmitted to the corresponding stage (or block) of the decoder (730), and each stage of the decoder (730) can perform a restoration operation by utilizing (e.g., concatenation) the low-level features transmitted from the encoder (710), thereby improving restoration performance.

[0119] According to one embodiment of the present disclosure, the object separation model (221) can be implemented as a model having a single input / output structure by being implemented as a model including at least one 1D convolution layer, at least one 1D-Tr convolution layer and / or LSTM layer.

[0120] According to one embodiment, the object separation model (221) may be a U-Net model. For example, as illustrated in FIG. 9, the object separation model (221) may be a U-Net model having skip connections (740) and a U shape. The U-Net model may have all and / or some of the characteristics of the object separation model (221) described above. For example, the U-Net model may have a single I / O structure feature, a feature including a 1D convolution layer and an LSTM layer, and / or a feature using a skip connection structure.

[0121] Hereinafter, with reference to FIG. 9, the operation of the object separation model (221) of the U-Net model is described in more detail as a non-limiting example. For convenience of explanation, the encoder and decoder are described as examples including five 1D convolution layers, but are not limited thereto.

[0122] Referring to FIG. 9, the object separation model (221) of the U-Net model can extract and compress key features on the frequency axis from input data in the frequency domain using an encoder comprising five 1D convolution layers. For example, as illustrated in FIG. 9, the encoder can compress on the frequency axis and generate high-level data by sequentially applying five 1D convolution layers corresponding to Convd (1,2) to input data of 2 x 1 x 512 (C x W x H) (wherein the input data corresponds to a complex number form, so the channel (C) is 2, the width (W) corresponds to one time step, and the height (H) corresponds to 512, so the number of frequency bins corresponds to 512). The encoded data can be reshaped and passed to an LSTM layer. The data that has passed through the LSTM layer can be reshaped and input to a decoder. The object separation model (221) of the U-Net model can restore encoded data using a decoder that includes five 1D-Tr convolution layers. In this case, the decoder can be used to restore low-level data transmitted from the encoder through skip connections. Through this, the decoder can output 6 x 1 x 512 (C x W x H) output data. The output data corresponds to mask data for three objects, and each mask data corresponds to a complex number form so that the channel (C) is 6 (=2*3), the width (W) corresponds to one time step so that the frequency bin count corresponds to 512 so that the height (H) is 512.

[0123] According to one embodiment, the object separation model (221) may be an on-device model stored in the memory of an electronic device (e.g., the memory (130) of FIG. 1).

[0124] An object separation model (221) having the above-described characteristics(s) can be easily implemented on a device and, for example, compared to a model that uses long past input data and includes a 2D convolution layer and a transformer, ensures real-time performance.

[0125] FIG. 10 is a drawing illustrating an exemplary method of using separated object audio data in an electronic device according to one embodiment of the present disclosure.

[0126] Referring to FIG. 10, an electronic device (200a) (e.g., electronic device (101) of FIG. 1, electronic device (200) of FIG. 2) can provide sound rendering using object audio data obtained through an object separation model (e.g., object separation model (221) of FIG. 2).

[0127] According to one embodiment, an electronic device (200a) may use object audio data to output sound suitable for content (e.g., video content) provided (e.g., displayed) by the electronic device (200a). For example, the electronic device (200a) (e.g., a display device such as a TV) may process (e.g., mixing) object audio data separated into voice, music, and sound effects so that they are output at different ratios to the electronic device (200a) and an external electronic device (200b) (e.g., a sound device such as a sound bar) connected to the electronic device (200a). For example, as illustrated in FIG. 10, when a fighter jet image is provided through an electronic device (200a), the electronic device (200a) can process the sound effect (S) of the fighter jet to be rendered in the direction of the fighter jet in the fighter jet image through the electronic device (200a) and / or an external electronic device (200b) connected to the electronic device (200a), thereby maximizing the sense of realism regarding the actual fighter jet. In this way, the electronic device (200a) can provide immersive sound to the user by utilizing separate object audio data.

[0128] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs from the description below.

[0129] Although the present disclosure has been illustrated and described with reference to various exemplary embodiments, it will be understood that these various exemplary embodiments are for illustrative purposes only and are not intended to be limiting. Furthermore, those skilled in the art will understand that various modifications, alternatives, and / or variations to the various exemplary embodiments may be made without departing from the true technical spirit and the entire technical scope of the present disclosure, including the appended claims and their equivalents. Furthermore, it will also be understood that any of the embodiment(s) described herein may be used in combination with other embodiment(s) described herein.

Claims

1. In an electronic device, Memory comprising at least one storage medium for storing instructions; and The electronic device comprises at least one processor including a processing circuit, wherein the at least one processor individually and / or collectively enables: To obtain input data for the object separation model, input audio data is preprocessed, and The above input data is input into the object separation model to obtain mask data for multiple objects, and It is configured to generate object audio data for the plurality of objects using mask data for the plurality of objects, and The operation of preprocessing the above input audio data is: The operation of converting the above input audio data into a frequency domain to obtain first frequency domain data having a first number of frequency bins; and An electronic device comprising an operation of converting the first frequency domain data into second frequency domain data having a second frequency bin count smaller than the first frequency bin count using a specified conversion algorithm, wherein the second frequency domain data is input to the object separation model as input data.

2. In Paragraph 1, The operation of converting to the above second frequency domain data is: The method includes applying a second frequency resolution higher than the first frequency resolution to frequency bins having a frequency greater than or equal to a reference frequency among the frequency bins of the first frequency domain data. An electronic device wherein the first frequency resolution is a frequency resolution applied to frequency bins having a frequency less than the reference frequency among the frequency bins of the first frequency domain data.

3. In Paragraph 1, The operation of converting to the above second frequency domain data is, The operation includes applying a specified transformation function to frequency bins having a frequency greater than or equal to a reference frequency among the frequency bins of the first frequency domain data. The above conversion function is set based on the reference frequency, the first frequency bin number, the second frequency bin number, and the first frequency resolution, and An electronic device in which the first frequency resolution is set to a value obtained by dividing the sampling rate of the input audio data by the size of the FFT (fast Fourier transform).

4. In Paragraph 2 or 3, An electronic device in which the above reference frequency is set based on the frequency band of a first object corresponding to voice among the plurality of objects.

5. In Paragraph 1, An FFT is used for the above frequency domain transformation, and The number of first frequency bins is half the size of the FFT of the above FFT, and An electronic device in which the second frequency bin is half the first frequency bin.

6. In Paragraph 1, The above object separation model corresponds to a U-Net model using a skip connection structure, and the above skip connection structure is a structure in which the output value of each stage of an encoder is connected to the corresponding stage of a decoder, an electronic device.

7. In paragraph 6, the object separation model is an electronic device having a single input / output structure.

8. An electronic device according to claim 6 or 7, wherein the object separation model comprises at least one 1D convolution layer, at least one 1D-Tr (transpose) convolution layer and an LSTM (long short-term memory) layer.

9. In any one of paragraphs 6 through 8, An electronic device wherein the object separation model includes an on-device model stored in the memory, and the time duration of the input audio data is set to a value smaller than 50ms.

10. In Paragraph 1, The operation of generating object audio data for the above plurality of objects is: An operation of recovering each mask data having the second frequency bin number using a specified recovery algorithm to obtain each recovered mask data having the first frequency bin number; and The method includes the operation of generating object audio data for each of the plurality of objects by multiplying the first frequency domain data by each of the restored mask data. The above plurality of objects comprises at least one of a first object corresponding to voice, a second object corresponding to music, or a third object corresponding to an effect, in an electronic device.

11. In a method of operating an electronic device, The operation of preprocessing input audio data to obtain input data for an object separation model; The operation of inputting the above input data into the object separation model to obtain mask data for a plurality of objects; and The method includes an operation of generating object audio data for the plurality of objects using mask data for the plurality of objects, and The operation of preprocessing the above input audio data is: The operation of converting the above input audio data into a frequency domain to obtain first frequency domain data having a first number of frequency bins; and A method comprising, using a specified conversion algorithm, converting the first frequency domain data into second frequency domain data having a second frequency bin number smaller than the first frequency bin number, wherein the second frequency domain data is input to the object separation model as input data.

12. In Paragraph 11, The operation of converting to the above second frequency domain data is: The method includes applying a second frequency resolution higher than the first frequency resolution to frequency bins having a frequency greater than or equal to a reference frequency among the frequency bins of the first frequency domain data. A method in which the first frequency resolution is a frequency resolution applied to frequency bins having a frequency less than the reference frequency among the frequency bins of the first frequency domain data.

13. In Paragraph 11, The operation of converting to the above second frequency domain data is, A method comprising applying a specified transformation function to frequency bins having a frequency greater than or equal to a reference frequency among the frequency bins of the first frequency domain data, wherein the transformation function is set based on the reference frequency, the number of first frequency bins, the number of second frequency bins, and the first frequency resolution, and the first frequency resolution is set as a value obtained by dividing the sampling rate of the input audio data by the size of the FFT (fast Fourier transform).

14. In Paragraph 12 or 13, A method in which the above reference frequency is set based on the frequency band of a first object corresponding to voice among the plurality of objects.

15. In Paragraph 11, An FFT is used for the above frequency domain transformation, and The number of first frequency bins is half the size of the FFT of the above FFT, and A method in which the second frequency bin number is half the first frequency bin number.

Images