Audio signal processing apparatus and method

Abstract

This audio signal processing apparatus may comprise: one or more processors; one or more memories for storing instructions; and a display for outputting an audio editing interface for editing an audio input signal. When the instructions are individually or collectively executed by the one or more processors, the audio signal processing apparatus can detect one or more target object audio signals from an audio input signal by using an object audio signal detection model having the audio input signal as an input, provide the audio editing interface on the display in response to the detection of the one or more target object audio signals, obtain the one or more target object audio signals separated from the audio input signal by using an object audio signal separation model having the audio input signal as an input, and generate an audio output signal by adjusting the audio signal level of the separated one or more target object audio signals in response to receiving an audio editing request input of a user through the audio editing interface.

Description

Audio signal processing device and method

[0001] The following disclosure relates to an audio signal processing device and method.

[0002] Sound source separation technology can refer to a technique for separating multiple sounds from a single audio signal (or audio file). Sound source separation can be performed by applying spectrum analysis techniques (e.g., spectrogram analysis of the sound source), frequency analysis techniques, or energy analysis techniques. Sound source separation technology can be utilized in various fields, for example, in music production through vocal removal, audio restoration with noise reduction, and speech recognition separated from background sound.

[0003] According to one embodiment, an audio signal processing device may include one or more processors, one or more memories for storing instructions, and a display for outputting an audio editing interface for editing an audio input signal. When instructions are executed individually or collectively by one or more processors, the audio signal processing device may detect one or more target object audio signals from the audio input signal using an object audio signal detection model that takes the audio input signal as input. In response to the detection of one or more target object audio signals, the audio signal processing device may provide an audio editing interface through the display. The audio signal processing device may obtain one or more target object audio signals separated from the audio input signal using an object audio signal separation model that takes the audio input signal as input. In response to receiving an audio editing request input from a user through the audio editing interface, the audio signal processing device may generate an audio output signal by adjusting the audio signal level of the one or more separated target object audio signals.

[0004] An audio signal processing method performed by an audio signal processing device according to one embodiment may include an operation of detecting one or more target object audio signals from an audio input signal using an object audio signal detection model that takes an audio input signal as input. The audio signal processing method may include an operation of providing an audio editing interface through a display in response to the detection of one or more target object audio signals. The audio signal processing method may include an operation of acquiring one or more target object audio signals separated from an audio input signal using an object audio signal separation model that takes an audio input signal as input. The audio signal processing method may include an operation of generating an audio output signal by adjusting the audio signal level of one or more separated target object audio signals in response to receiving an audio editing request input from a user through the audio editing interface.

[0005] The aspects mentioned above and other aspects, features, and advantages of specific embodiments of the present disclosure will become more apparent from the following detailed description together with the accompanying drawings:

[0006] FIG. 1 is a block diagram of an audio signal processing device in a network environment according to one embodiment.

[0007] FIG. 2 is a schematic diagram illustrating an audio signal processing device according to one embodiment detecting and separating an object audio signal.

[0008] FIG. 3 is a flowchart illustrating an audio signal processing method according to one embodiment.

[0009] FIG. 4 is a flowchart illustrating the detection of a target object audio signal using a classification-based model according to one embodiment.

[0010] FIG. 5 is a diagram illustrating the detection of a target object audio signal using a classification-based model according to one embodiment.

[0011] FIG. 6 is a flowchart illustrating the detection of a target object audio signal using a lightweight sound source separation model according to one embodiment.

[0012] FIG. 7 is a diagram illustrating the detection of a target object audio signal using a lightweight sound source separation model according to one embodiment.

[0013] FIGS. 8A and FIGS. 8B are drawings for illustrating an interface provided in an audio signal processing device according to one embodiment.

[0014] FIG. 9 is a diagram illustrating an example in which an audio signal processing device is used in an image according to one embodiment.

[0015] FIG. 10 is a block diagram illustrating the configurations of an audio signal processing device according to one embodiment.

[0016] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified and implemented in various forms. Accordingly, actual implementations are not limited to the specific embodiments disclosed, and the scope of this specification includes modifications, equivalents, or substitutions included in the technical concept described by the embodiments.

[0017] Terms such as "first" or "second" may be used to describe various components, but these terms should be interpreted solely for the purpose of distinguishing one component from another. For example, the first component may be named the second component, and similarly, the second component may be named the first component.

[0018] When it is stated that a component is "connected" to another component, it should be understood that it may be directly connected to or joined to that other component, or that there may be other components in between.

[0019] Singular expressions include plural expressions unless the context clearly indicates otherwise. In this document, phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B or C,” “at least one of A, B and C,” and “at least one of A, B, or C” may each include any one of the items listed together with the corresponding phrase, or all possible combinations thereof. In this specification, terms such as “comprising” or “having” are intended to designate the existence of the described features, numbers, steps, actions, components, parts, or combinations thereof, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0020] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this specification.

[0021] As used herein, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0022] As used in this document, the term "part" refers to a software or hardware component, such as an FPGA or ASIC, that performs certain roles. However, "part" is not limited to software or hardware. "Part" may be configured to reside in an addressable storage medium or configured to operate one or more processors. For example, "part" may include components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and "parts" may be combined into a smaller number of components and "parts" or further separated into additional components and "parts." Furthermore, components and "parts" may be implemented to operate one or more CPUs within a device or secure multimedia card. Additionally, '~part' may include one or more processors.

[0023] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are given the same reference numeral regardless of the drawing number, and redundant descriptions thereof will be omitted.

[0024]

[0025] FIG. 1 is a block diagram of an audio signal processing device in a network environment according to one embodiment.

[0026] Referring to FIG. 1, in a network environment (100), an audio signal processing device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the audio signal processing device (101) may communicate with an electronic device (104) through a server (108). According to one embodiment, the audio signal processing device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the audio signal processing device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0027] The processor (120) can, for example, execute software (e.g., program (140)) to control at least one other component (e.g., hardware or software component) of the audio signal processing device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the audio signal processing device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0028] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the audio signal processing device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the audio signal processing device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0029] The memory (130) may store various data used by at least one component of the audio signal processing device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, software (e.g., program (140)) and input data or output data for related instructions. The memory (130) may include volatile memory (132) or non-volatile memory (134). The memory (130) may be a computer-readable storage medium for storing instructions, and when the instructions stored in the memory (130) are executed by the processor (120), the audio signal processing device (101) may prompt at least one processor (120) to perform an audio signal processing method.Computer-readable storage media include read-only memory (ROM), programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, BLU-RAY, or optical disc memory, hard disk drives (HDD), solid-state hard disks (SSD), card memory (e.g., multimedia cards, secure digital (SD) cards, or extreme digital (XD) cards), magnetic tape, floppy disks, It may include magneto-optical data storage devices, optical data storage devices, hard disks, solid-state disks, and other devices.

[0030] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0031] The input module (150) can receive commands or data to be used for a component (e.g., processor (120)) of the audio signal processing device (101) from outside the audio signal processing device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0032] The sound output module (155) can output a sound signal to the outside of the audio signal processing device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0033] The display module (160) can visually provide information to an external (e.g., user) of the audio signal processing device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0034] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the audio signal processing device (101).

[0035] The sensor module (176) can detect the operating state of the audio signal processing device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0036] The interface (177) may support one or more specified protocols that can be used for the audio signal processing device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0037] The connection terminal (178) may include a connector through which the audio signal processing device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0038] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0039] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0040] The power management module (188) can manage the power supplied to the audio signal processing device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0041] The battery (189) can supply power to at least one component of the audio signal processing device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0042] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an audio signal processing device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate an audio signal processing device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0043] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified by the audio signal processing device (101), an external electronic device (e.g., electronic device (104)), or a network system (e.g., a second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0044] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0045] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0046] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0047] According to one embodiment, commands or data may be transmitted or received between the audio signal processing device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or different type of device as the audio signal processing device (101). According to one embodiment, all or part of the operations performed in the audio signal processing device (101) may be performed in one or more of the external electronic devices (102, 104, or 108). For example, if the audio signal processing device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the audio signal processing device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the audio signal processing device (101). The audio signal processing device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The audio signal processing device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks.According to one embodiment, an external electronic device (104) or server (108) may be included within the second network (199). The audio signal processing device (101) may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0048]

[0049] FIG. 2 is a schematic diagram illustrating an audio signal processing device according to one embodiment detecting and separating an object audio signal.

[0050] Referring to FIG. 2, the audio signal processing device is a device that edits an audio input signal (201) and outputs an audio output signal (203). The audio input signal (201) may represent voice data obtained through a microphone or obtained from a voice file. For example, the audio input signal (201) may represent an audio signal containing a human voice input through a microphone and background music, or an audio signal in which the sounds of various instruments stored in a music file are combined. However, the audio input signal (201) is not limited thereto and may include a single sound or a combination of multiple sounds. The audio signal processing device may include a two-step process to reduce the time used to edit the audio input signal (201). The first step is to detect an object audio signal included in the audio input signal (201). The object audio signal may represent a meaningful object sound source or a sound source of interest that is the subject of individual editing in the audio input signal (201), but is not limited thereto. Object audio signal detection may be referred to in various ways, such as object audio signal scanning or object audio signal classification, and is not limited thereto. The second step is to separate the detected object audio signal from the audio input signal (201). The term 'separation' may be replaced with terms such as extraction or filtering, but is not limited thereto.

[0051] The audio signal processing device can detect object audio signals from an audio input signal (201) and provide information regarding the detected object audio signals through a user interface (220). The detected object audio signals are separated from the audio input signal (201) in real time (e.g., as soon as each object audio signal is detected) and can be edited (e.g., increase volume or adjust other audio characteristics) by inputting an audio editing request (202) from the user through the user interface (220). Since there is no need to wait until all object audio signals included in the audio input signal (201) are detected according to the above process, the audio signal processing device can reduce the waiting time for editing object audio signals. Furthermore, this method means that the user does not need to wait until all target object audio signals are separated in order to check or edit the target object audio signals. In some examples, the detection of target object audio signals is performed separately from the separation of target object audio signals and is performed faster than the separation of object audio signals, so the result of the detection of target object audio signals can be provided to the user while the separation of target object audio signals is in progress. In some examples, the separation of the target object audio signal can be performed only for the segments of the audio input signal where the target object audio signal is detected.

[0052] In other examples, the detection of target object audio signals, which is generally a lower complexity operation than the separation of target object audio signals, can be performed offline on the audio signal processing unit (e.g., using a local model), whereas the separation of target object audio signals can be performed online (e.g., using a server or cloud-based model). This approach enables rapid detection of target object audio signals without significantly increasing resource usage on the audio signal processing unit, whereas the more resource-intensive separation of target object audio signals can be offloaded to reduce resource usage on the audio signal processing unit. This approach can be particularly advantageous for resource-constrained electronic devices, including battery-powered smartphones.

[0053] The audio signal processing device can perform an object audio signal detection operation (210) on an audio input signal (201). The audio signal processing device can detect one or more target object audio signals (e.g., human conversation sound and wind sound) from the audio input signal (201) using an object audio signal detection model that takes the audio input signal (201) as input. The object audio signal detection model may be a classification-based model or a lightweight / complexity-reduced sound source separation model (e.g., the sound source separation model (710) of FIG. 7). A description of the object audio signal detection model is explained in more detail in FIG. 5 and FIG. 7.

[0054] The audio signal processing device may provide a user interface (220) for the detected target object audio signal. For example, the audio signal processing device may provide an audio editing interface via a display in response to a user's request to detect one or more target object audio signals or to detect and / or separate one or more target object audio signals from an audio input signal. A user's audio editing request input (202) may be input through the audio editing interface. The user's audio editing request input (202) may include a request to adjust the size (e.g., volume adjustment) of the audio signal level of the detected target object audio signal. The user's audio editing request input (202) may be input through an interface, or through text input or voice signal input (e.g., "Adjust the volume of the song to 60% of the existing volume"). The audio signal level may indicate the magnitude of the volume of the audio signal. The audio editing interface may provide information about the detected target object audio signal and may include a volume level control interface for adjusting the magnitude of the level of the detected target object audio signal. The description of the audio editing interface is explained in more detail in Figures 8a and 8b.

[0055] The audio signal processing device can perform an object audio signal separation (230) operation on an audio input signal (201). For example, the audio signal processing device can obtain one or more target object audio signals separated from the audio input signal (201) by using an object audio signal separation (230) model that takes the audio input signal (201) as input. In one embodiment, the audio signal processing device can obtain the separated target object audio signals in real time after detecting the target object audio signals from the audio input signal (201). For example, the audio signal processing device can obtain the separated target object audio signals for the entire frame in real time by detecting the target object audio signals from the audio input signal (201) input for each frame and obtaining the target object audio signals separated for each frame from the object audio signal separation (230) model.

[0056] An audio signal processing device according to one embodiment may generate an audio output signal (203) in response to receiving a user's audio editing request input (202) through a user interface (220) (e.g., an audio editing interface). For example, the audio signal processing device may generate the audio output signal (203) by adjusting the audio signal levels of one or more separated target object audio signals. The audio signal processing device may generate the output signal using an audio mixer (240). An audio mixer is a device that mixes audio signals and outputs a mixed audio signal. The audio signal processing device may obtain an audio output signal (203) from an audio mixer (240) that has received an audio input signal (201), separated target object audio signals, and a user's audio editing request input (202). The audio output signal (203) may represent an audio signal in which the volume level of the target object audio signal in the audio input signal (201) has been increased or decreased.

[0057]

[0058] FIG. 3 is a flowchart illustrating an audio signal processing method according to one embodiment.

[0059] The operations of the audio signal processing method can be performed by an audio signal processing device (e.g., the audio signal processing device (101) of FIG. 1).

[0060] In operation (310), the audio signal processing device may receive an audio input signal (e.g., the audio input signal (201) of FIG. 2). For example, the audio signal processing device may receive an audio input signal that combines the conversational voices of multiple people, background music, and noise through a microphone (not shown) connected via an input module (e.g., the input module (150) of FIG. 1).

[0061] In operation (320), the audio signal processing device can detect a target object audio signal. The audio signal processing device can detect one or more target object audio signals from an audio input signal using an object audio signal detection model. For example, the audio signal processing device can input the audio input signal into an object audio signal detection model and obtain output values ​​for conversational speech, background music, and noise from the object audio signal detection model. The audio signal processing device can determine whether to detect conversational speech, background music, and noise in the audio input signal using each of the obtained values. The detection of a target object audio signal by the audio signal processing device using an object audio signal detection model that takes the audio input signal as input is explained in more detail in FIGS. 5 and 7.

[0062] In operation (330), the audio signal processing device may provide an audio editing interface. The audio editing interface may represent an interface for adjusting the audio signal level (or volume) of a target object audio signal detected in an audio input signal. The audio signal processing device may provide an audio editing interface via a display in response to the detection of one or more target object audio signals. For example, if the audio signal processing device detects dialogue voice, background music, and noise in the audio input signal, the audio signal processing device may provide an interface for adjusting the volume of the dialogue voice, an interface for adjusting the volume of the background music, and an interface for adjusting the volume of the noise. The interface may be displayed in response to the start of the detection of the target object audio signal or in response to the actual detection of the target object audio signal in the audio input signal. The audio editing interface may provide interval information for each of the detected dialogue voice, background music, and noise (e.g., in which time interval of the audio input signal the target object audio signal is contained), or may include a waveform interface that represents each detected sound as a waveform.

[0063] In operation (340), the audio signal processing device may obtain a target audio signal level. The target audio signal level may represent an audio signal level requested by the user for the detected target object audio signal. For example, if the detected target object audio signal is noise, the user may request muting for the noise. In this case, the target audio signal level obtained by the audio signal processing device may be 0%. If the detected target object audio signal is interesting to the user, the level of the target audio signal obtained by the audio signal processing device may be, for example, 150%.

[0064] In operation (350), the audio signal processing device can separate and acquire target object audio signals. The audio signal processing device can acquire one or more target object audio signals separated from the audio input signal using an object audio signal separation model that takes an audio input signal as input. For example, the audio signal processing device can detect conversational speech in an input frame and then acquire conversational speech separated from the audio input signal using an object audio signal separation model. The audio signal processing device can detect noise in an input frame and then acquire noise separated from the audio input signal using an object audio signal separation model.

[0065] The audio signal processing unit can separate the detected target object audio signal from the audio input signal when the target object audio signal is detected in the input frame, without waiting to see if other object audio signals have been detected. Through the above process, the audio signal processing unit can perform the operation of separating the target object audio signal from the audio input signal in real time. By performing the operation of separating the target object audio signal in real time, the audio signal processing unit can reduce the waiting time required for the user to edit the target object audio signal. For example, when the target object audio signal is detected, the audio signal processing unit can perform the task of separating the target object audio signal from the audio input signal instead of waiting until detection is complete for the entire audio input signal. Additionally, the user can apply desired adjustments to the target object audio signal even while it is still being separated, thereby further reducing the delay for the user in making adjustments.

[0066] In operation (360), the audio signal processing device can adjust the audio signal level of the target object audio signal to the target audio signal level. The audio signal processing device can adjust the audio signal level of one or more separated target object audio signals in response to receiving an audio editing request input from a user. The audio signal processing device can adjust the audio signal level by adjusting a weight for the target object audio signal. The weight may represent a value included in a defined range (e.g., a value from 0 to 2).

[0067] According to one embodiment, a user can individually adjust the audio signal level for a detected target object audio signal through a user interface (e.g., the user interface (220) of FIG. 2). For example, the detected target object audio signal may be conversational voice, background music, and noise, and the audio signal processing device may provide an audio editing interface for each. The user can individually adjust the audio signal levels of conversational voice and noise by adjusting the weight for conversational voice to 1.5 through the audio editing interface for conversational voice and adjusting the weight for noise to 0 through the audio editing interface for noise.

[0068] When an audio signal processing device receives an audio editing request input that corresponds to an automatic adjustment request for adjusting the audio signal level of each target object audio signal separated from the audio input signal, the device can adjust the audio signal level of each detected target object audio signal to a preset value (or preset value).

[0069] When an audio editing request input including an automatic adjustment request is input to an audio signal processing device using a classification-based model according to one embodiment, the audio signal processing device may adjust the audio signal level of each separated target object audio signal to a level defined for each. Adjusting the audio signal level of each separated target object audio signal to a level defined for each may be referred to as a fixed preset or a fixed preset mode. For example, if each separated target object audio signal is a conversational voice, background music, and noise, the audio signal processing device using the classification-based model may adjust the volume of the conversational voice to 100%, the volume of the background music to 0%, and the volume of the noise to 0%.

[0070] When an audio editing request input including an automatic adjustment request is input to an audio signal processing device using a lightweight sound source separation model according to one embodiment, the audio signal level of each separated target object audio signal can be adjusted to a level determined by the ratio of the size of the target object audio signal to the size of the audio input signal. Adjusting the audio signal level of each separated target object audio signal to a level determined by the ratio of the size of the target object audio signal to the size of the audio input signal may be referred to as an adaptive preset, an adaptive preset mode, or a variable preset mode. The level determined by the ratio of the size of the target object audio signal to the size of the audio input signal can be defined by the following Equation 1.

[0071]

[0072] In Equation 1, B represents the magnitude of the detected target object audio signal, and A represents the magnitude of the audio input signal. The unit of Equation 1 is dB.

[0073] The audio signal processing device can determine a level according to an adaptive preset mode for each of one or more target object audio signals detected in an audio input signal. For example, if each of the separated target object audio signals is conversational voice, background music, and noise, the level according to the preset mode for conversational voice can be adjusted to 30 dB, the level according to the preset mode for background music can be adjusted to 17 dB, and the level according to the preset mode for noise can be adjusted to 0 dB.

[0074]

[0075] FIG. 4 is a flowchart illustrating the detection of a target object audio signal using a classification-based model according to one embodiment.

[0076] Referring to FIG. 4, in operation (410), an audio signal processing device (e.g., the audio signal processing device (101) of FIG. 1) can receive an audio input signal. Since the receiving of an audio input signal by the audio signal processing device has been described in detail in FIG. 3, a redundant description will be omitted.

[0077] In operation (420), the audio signal processing device can detect sound events using an object audio signal model. The object audio signal detection model may be a classification-based model that estimates whether a target object audio signal is detected in an audio input signal based on sound event detection. Sound event detection may involve identifying a specific sound (e.g., conversational voice, noise, clapping sound, dog barking sound, etc.) in an audio input signal and detecting the time and duration at which the specific sound occurs. The audio signal processing device can estimate whether a target object audio signal is detected in an audio input signal using a classification-based model trained to detect target object audio signals (e.g., conversational voice, noise, background music).

[0078] In operation (430), the audio signal processing device can determine whether the output value of the classification-based model is greater than a first threshold value. If the output value for the target object audio signal obtained from the object audio signal detection model is greater than the first threshold value, the audio signal processing device can determine that the target object audio signal has been detected from the audio input signal. The output value for the target object audio signal may correspond to a probability value of the existence of the target object audio signal in the audio input signal. According to one embodiment, the first threshold value may be 0.75 and the output value may be 0.8. Since the output value 0.8 is greater than the first threshold value 0.75, the audio signal processing device can determine that conversational voice has been detected. In one embodiment, the first threshold value may be 0.75 and the audio signal processing device may determine that the output value 0.64 is the value at which conversational voice is estimated to have been detected from the classification-based model. Since the output value 0.64 is smaller than the first threshold value 0.75, the audio signal processing device can determine that the detected target object audio signal is not conversational voice. Detecting target object audio signals using a classification-based model will be explained in detail in Fig. 5.

[0079] In operation (440), the audio signal processing device may provide information about the detected target object audio signal through an interface. For example, the audio signal processing device may provide interval information indicating the time interval in which one or more target object audio signals detected from the audio input signal were detected through an audio editing interface. The audio interface providing interval information through the interface is described in detail in FIG. 8a.

[0080]

[0081] FIG. 5 is a diagram illustrating the detection of a target object audio signal using a classification-based model according to one embodiment.

[0082] Referring to FIG. 5, an audio signal processing device (e.g., the audio signal processing device (101) of FIG. 1) can detect one or more target object audio signals using an object audio signal detection model that takes an audio input signal (201) as input. The object audio signal detection model may be a classification-based model that estimates whether a target object audio signal is detected in the audio input signal (201) based on sound event detection.

[0083] A classification-based model (510) can classify one or more target object audio signals using similarity and distance relationships for object audio signal data included in the audio input signal (201). For example, the classification-based model (510) may be based on a deep neural network (DNN), a support vector machine (SVM), and / or a k-nearest neighbors (KNN) algorithm. The classification-based model (510) can detect sound events using unique features of the target object audio signals (e.g., speech, music, wind sound, siren sound, animal sound, etc.). The classification-based model (510) can detect target object audio signals using the results of detecting sound events and classify the detected results. The classification-based model (510) can output one or more of the following: each target object audio signal estimated to have been detected using the results of classifying the target object audio, an output value (e.g., probability value) for the estimated target object audio signal, and a time of occurrence.

[0084] The audio signal processing device may determine that one or more target object audio signals have been detected by using a classification-based object audio signal detection model that takes an audio input signal (201) as input. The audio signal processing device may determine that a target object audio signal has been detected from the audio input signal (201) if the output value of the target object audio signal obtained from the object audio signal detection model is greater than a first threshold value. Since the determination of a target object audio signal by the audio signal processing device has been explained in detail in FIG. 4, a redundant explanation will be omitted.

[0085] The audio signal processing device can determine whether one or more target object audio signals are detected per frame using result values ​​obtained from a classification-based object audio signal detection model. The graph (520) shows the conversation (520d), wind (520c), music (520b), and other (520a) sounds detected in the first frame (521), second frame (522), and third frame (523) sections. The audio signal processing device can determine that music (520b), wind (520c), and conversation (520d) sounds are detected in the first frame (521) section using values ​​output from the classification-based model (510), determine that conversation (520d), music (520b), and other sounds (520a) are detected in the second frame (522) section, and determine that conversation (520d), music (520b), and other sounds (520a) are detected in the third frame (523) section. When detecting a target object audio signal using a classification-based object audio signal detection model, the detection time can be reduced compared to when using a sound source separation model or a lightweight sound source separation model. Since the classification model-based object audio signal detection model provides only approximate information including the type of the detected target object audio and the section where the target object audio exists, the time used for detecting the target object audio signal can be reduced compared to a sound source separation model or a lightweight sound source separation model. The frame duration can be any appropriate value and can improve the accuracy and / or speed of target object audio signal detection. In some examples, where the audio input signal corresponds to video, the frame duration can be the frame length of the video.

[0086] The classification-based model (510) can be trained through a supervised learning method. The process of training the classification-based model (510) may include, for example, preprocessing training data, detecting a target object audio signal predicted by the classification-based model (510) using the preprocessed training data, and updating the parameters of the classification-based model (510) using the detected target object audio signal.

[0087] The training data used in the classification-based model (510) may include training audio input signals and label data. The process of preprocessing the training data may include a normalization process for adjusting the amplitude of the training audio signal, a sampling process for the training audio signal, and a noise removal process. The sampling process for the training audio signal may involve digitizing the input training audio source signal if it is an analog signal. The data format of the input training audio signal may be converted through the preprocessing process into a data format (e.g., latent representation) that can be used more effectively in an object audio detection model.

[0088] The process of detecting a target object audio signal detected from a classification-based model (510) is a process of detecting a target object audio signal through a process of encoding and decoding a preprocessed learning audio input signal. The classification-based model (510) can output a result of detecting a sound event through a process of encoding and decoding the learning audio input signal.

[0089] The process of optimizing the classification-based model (510) may include determining a loss (or loss function) for a predicted value output from the classification-based model (510) and minimizing the determined loss. The process of minimizing the determined loss may include differentiating the loss function to determine the extent to which each parameter of the classification-based model (510) contributed to the loss, and updating the parameters according to the degree of contribution. Gradient descent or a modified method thereof may be used for updating the parameters. Through this learning process, the classification-based model (510) may learn patterns from the learning audio input signal (201) and have the ability to detect target object audio signals from new audio input signals.

[0090]

[0091] FIG. 6 is a flowchart illustrating the detection of a target object audio signal using a sound source separation model according to one embodiment.

[0092] Referring to FIG. 6, in operation (610), an audio signal processing device (e.g., the audio signal processing device (101) of FIG. 1) can receive an audio input signal. Since the receiving of an audio input signal by the audio signal processing device has been described in detail in FIG. 3, a redundant description will be omitted.

[0093] In operation (620), the audio signal processing device may preprocess the audio input signal. For example, the audio signal processing device may downsample the audio input signal as a preprocess. Downsampling may refer to reducing the data size of the audio input signal. The audio signal processing device may reduce the size of the audio input signal by performing downsampling, and consequently, downsampling may increase the detection speed during the process of detecting the target object audio signal.

[0094] In the process of detecting target object audio signals, the data size of the audio input signal can be reduced by performing downsampling to increase detection speed. Downsampling can be performed by passing the audio input signal through a digital filter (e.g., a low-pass filter) or by resampling the audio input signal. Although downsampling has been mentioned, any other preprocessing techniques may be used to increase detection speed or reduce complexity when using a source separation model so that the detection of the target object audio signal is faster than its separation. In some examples, a lightweight version of the source separation model may be used for the detection of target object audio signals as an addition to or alternative to preprocessing.

[0095] In operation (630), the audio signal processing device may obtain a target object audio signal presumed to have been detected from a lightweight sound source separation model. The sound source separation model may represent an object audio signal detection model that takes an audio input signal as input and outputs one or more target object audio signals presumed to have been detected from the audio input signal. The one or more target object audio signals presumed to have been detected output from the sound source separation model may not be target object audio signals separated from the audio input signal, but may represent approximate audio signals for the target object audio signals presumed to have been detected. The outputting of approximate audio signals for the target object audio signals presumed to have been detected by the sound source separation model is described in detail in FIG. 7.

[0096] In operation (640), the audio signal processing device may determine whether the magnitude value of the target object audio signal is greater than a second threshold value. The magnitude value of the audio signal may be expressed in decibels (dB). For example, the audio signal processing device may determine whether the magnitude of the detected target object audio signal obtained from the sound source separation model is a sound exceeding 7 dB. If the magnitude value of the audio signal of the target object audio signal is greater than the second threshold value (e.g., 7 dB), the audio signal processing device may determine that the target object audio signal has been detected from the audio input signal. The audio signal processing device may determine whether the magnitude of each of the detected target object audio signals is greater than the second threshold value. The audio signal processing device may determine that each of the target object audio signals that is greater than the second threshold value has been detected. The second threshold value (7 dB) is one embodiment and is not limited thereto.

[0097] In operation (650), the audio signal processing device may provide a waveform for the detected target object audio signal through an interface. For example, the audio signal processing device may provide a waveform interface (e.g., the waveform interface (850) of FIG. 8b) that represents the detected target object audio signal as a waveform through an audio editing interface.

[0098] In operation (660), the audio signal processing device may provide information about the detected target object audio signal through an interface. For example, the audio signal processing device may provide the type of one or more target object audio signals detected in the audio input signal (e.g., the sound source type interface (840) of FIG. 8b) through an audio editing interface.

[0099]

[0100] FIG. 7 is a diagram illustrating the detection of a target object audio signal using a sound source separation model according to one embodiment.

[0101] Referring to FIG. 7, an audio signal processing device (e.g., the audio signal processing device (101) of FIG. 1) can detect one or more target object audio signals using an object audio signal detection model that takes an audio input signal (201) as input. The object audio signal detection model may be a sound source separation model (710) that takes an audio input signal as input and outputs one or more target object audio signals presumed to be detected from the audio input signal, and the sound source separation model (710) may include a lightweight model of the object audio signal separation model. The lightweight model of the object audio signal separation model may represent a machine learning model that includes the same structure (or a similar structure) as the structure (e.g., type and number of layers) of the object audio signal separation model used to separate the object audio signal detected from the audio input signal (201), and reduces the size (e.g., number of parameters included in each layer) of the object audio signal separation model. If the sound source separation model (710) (e.g., detection model) is a lightweight model of the object audio signal separation model (e.g., separation model), the data input to the sound source separation model (710) and the object audio signal separation model may be similar. For example, the data input to the sound source separation model (710) and the data input to the object audio signal separation model are of the same type, but the size of the data input to the sound source separation model (710) may be smaller than the size of the data input to the object audio signal separation model (e.g., by applying appropriate preprocessing described above). Alternatively, the input data may be the same, and the lightweight / reduced complexity characteristics of the sound source separation model may cause the speed at which the target object audio signal is detected to be faster compared to the speed at which the target object audio signal is separated.In some examples, to further reduce complexity or to further improve detection speed, the preprocessed data can be used as input to a lightweight / complexity-reduced sound source separation model (710) that performs detection.

[0102] The layer structure and types of the sound source separation model (710) may be similar to the layer structure and types of the object audio signal separation model. For example, the types of layers included in the sound source separation model (710) and the types of layers included in the object audio signal separation model are the same, but the number of layers included in each layer (e.g., the number of layers included in each hidden layer) may be different. The sound source separation model (710) may have a faster data processing speed compared to the object audio signal separation model due to its lightweight nature. Additionally, when the sound source separation model (710) is used, the mismatch phenomenon may be reduced. The mismatch phenomenon may manifest as a phenomenon where the target object audio signal is detected in the object audio detection stage but is not separated in the target object audio signal separation stage, or a phenomenon where the target object audio signal is not detected in the detection stage but is separated in the target object audio separation stage. The sound source separation model (710) uses input data similar to the object audio signal separation model, and since the structure and type of the included layers are similar, the output value of the sound source separation model (710) and the output value of the object audio signal separation model may be similar. By training the sound source separation model (710) with the training target object audio signal used to train the object audio signal separation model, the sound source separation model (710) can output a value similar to the output value of the object audio signal separation model. The similarity of the output value may indicate that the difference between the output value of the object audio signal separation model and the output value of the sound source separation model (710) is reduced compared to the existing one. When the difference between the output value of the sound source separation model (710) and the output value of the object audio signal separation model is reduced, the mismatch phenomenon may be reduced. The sound source separation model (710) is not limited to a lightweight model of the object audio signal detection model, but may include other machine learning models capable of detecting a target object audio signal from an audio input signal (210).

[0103] The sound source separation model (710) may be a machine learning model based on a convolutional neural network (CNN), a recurrent neural network (RNN), or a convolutional recurrent neural network (CRNN), which is a hybrid model of a convolutional neural network and a recurrent neural network. The process by which the sound source separation model (710) detects a target object audio signal from an audio input signal (201) may be similar to the process of classifying objects in an image. The sound source separation model (710) may include an input layer, a hidden layer, and an output layer. The dimension of the data input through the input layer may be changed to a dimension of data that the machine learning model can utilize effectively. For example, when an audio input signal (201) is input, the audio input signal (201) is converted into a spectrogram image (e.g., a spectrogram image of size 28 x 28) which is 2-dimensional data, and the converted image data can be converted into a 1-dimensional vector (e.g., a vector having 784 components of 1 dimension). The hidden layer can extract features for the input data and output a target object audio signal presumed to be detected using the extracted features. The hidden layer may include multiple layers for feature extraction, and each of the multiple layers may include multiple nodes. Each node is assigned a weight, and the corresponding weight can be optimized through a learning process. The hidden layer can output a target object audio signal presumed to be detected through a fully connected layer in which the input node and the output node are fully connected. The output layer can provide a target object audio signal presumed to be detected using the result value received from the hidden layer.For example, the output layer can provide an audio signal reconstructed from the target object audio signal presumed to have been detected, and a schematic waveform of the reconstructed audio signal. Through the above process, the object audio signal detection model can estimate whether a target object signal has been detected from the input audio source signal.

[0104] A sound source separation model (710) that receives an audio input signal (201) can output one or more target object audio signals presumed to be detected. For example, the sound source separation model (710) can output a voice audio signal (721), a wind audio signal (722), a music audio signal (723), an animal audio signal, and other audio signals (724) presumed to be detected. The sound source separation model (710) can determine that a target object audio signal is detected if the magnitude of each audio signal is greater than a second threshold value. The second threshold value may vary for each target object audio presumed to be detected. For example, the second threshold value for the voice audio signal (721) may be 15 dB, the second threshold value for the wind audio signal (722) may be 10 dB, the second threshold value for the music audio signal (723) may be 12 dB, the second threshold value for the animal audio signal may be 11 dB, and the second threshold value for other audio signals (724) may be 7 dB. The sound source separation model (710) can estimate that the voice audio signal (721), wind audio signal (722), music audio signal (723), and other audio signal (724) have been detected when the magnitude of the voice audio signal (721) is 17 dB, the magnitude of the wind audio signal (722) is 12 dB, the magnitude of the music audio signal (723) is 14 dB, the magnitude of the animal audio signal is 7 dB, and the magnitude of the other audio signal (724) is 9 dB. The sound source separation model (710) can provide the type of one or more target object audio signals estimated to have been detected and the waveform for the target object audio estimated to have been detected.For example, the sound source separation model (710) may provide voice (731), wind (732), music (733) and other (734) as types of target object audio presumed to be detected, and may provide a waveform (731a) corresponding to voice, a waveform (732a) corresponding to wind, a waveform (733a) corresponding to music, and a waveform (734a) corresponding to other.

[0105] The sound source separation model (710) can be trained using a supervised learning method similar to the training method used to train a classification-based model. The process of training the sound source separation model (710) may include preprocessing training data, detecting a target object audio signal predicted by the sound source separation model (710) using the preprocessed training data, and updating the parameters of the sound source separation model (710) using the detected target object audio signal. The training data used for the sound source separation model (710) may include training audio input signals and label data. The process of preprocessing the training data may include a normalization process for adjusting the amplitude of the training audio signal, a sampling process for the training audio signal, a visualization process for the sampled training audio signal, and a noise removal process. The sampling process for the training audio signal may involve digitizing the input training audio signal if it is an analog signal. The process of visualizing the sampled training audio signal may involve representing the audio signal information as a two-dimensional image. For example, a Fourier transform (e.g., Fast Fourier Transform) can be performed on a sampled audio signal to convert the input training audio signal into a two-dimensional image representing frequency features. The data format of the input training audio signal can be converted through a preprocessing process into a data format (e.g., latent representation) that can be used more effectively in an object audio detection model.

[0106] The process of detecting the target object audio signal detected from the sound source separation model (710) is a process of detecting the target object audio signal through the process of encoding and decoding the preprocessed learning audio input signal. Features can be extracted by encoding the learning audio input signal, and the target object audio signal presumed to be detected can be output by decoding using the extracted features.

[0107] The process of optimizing the sound source separation model (710) may include determining a loss (or loss function) for the predicted value output from the sound source separation model (710) and minimizing the determined loss. Since the process of minimizing the determined loss is the same as the process of minimizing the loss in the student separation model, a redundant explanation will be omitted. Through this learning process, the sound source separation model (710) may learn patterns from the learning audio input signal and have the ability to detect target object audio signals from the new audio input signal (201).

[0108]

[0109] FIGS. 8A and 8B are drawings for illustrating interfaces provided in an audio signal processing device according to one embodiment. The interfaces of FIGS. 8A and 8B may be provided in response to a user's request for target object audio signal detection, and the user's request may be provided through the preceding user interface screen.

[0110] Referring to FIG. 8a, when the audio signal processing device uses a classification-based object audio signal detection model, it may provide an audio editing interface for editing audio input signals. The audio editing interface may include an interface (820) for providing segment information indicating a segment of the audio input signal in which a target object audio signal presumed to be detected is detected, an interface (840) for displaying one or more detected target object audio signals, an automatic adjustment interface (830) for automatically adjusting the audio signal level of the separated target object audio signals, and an interface (810) for a user to individually adjust the size of the audio signal level of the separated target object audio signals. The interface (820) for displaying segment information may provide information regarding the segment (821) in which the detected target object audio signal is detected within the entire segment of the audio input signal. The interface (840) for displaying one or more detected target object audio signals may display each detected target object audio signal individually. For example, a conversational voice interface (841), a music interface (842), and a noise interface (843) may be displayed.

[0111] According to one embodiment, a user can adjust the audio signal level of a separated target object audio signal using a preset mode or adjust it individually. For example, if the user activates a fixed preset mode through the automatic adjustment interface (830), each audio signal level of the separated target object audio signal can be adjusted to a defined level. If the user wishes to adjust the separated target object audio signal individually, the user can select one of the conversational voice interface (841), the music interface (842), or the noise interface (843). The user can adjust the audio signal level of the target object audio signal by sliding the user inputs on the display in the first direction (811) or the second direction (812) in the interface (810) for individually adjusting the audio signal level, or by selecting the mute interface (813). Since adjusting the audio signal level is described in detail in FIG. 3, a redundant description will be omitted.

[0112] Referring to FIG. 8b, when the audio signal processing device uses a lightweight sound source separation model, it may provide an audio editing interface for editing audio input signals. Unlike when the audio signal processing device uses a classification-based model for object audio detection, when the audio signal processing device uses a lightweight sound source separation model, it may provide a waveform interface (850) for target object detection audio signals. The audio signal processing device may provide the waveform and magnitude of the detected target object audio signal in a schematic manner through the waveform interface (850). When the audio signal processing device uses a lightweight sound source separation model, the user individually adjusting the detected target object audio signals is the same as described in FIG. 8a, so a redundant description is omitted. According to one embodiment, the user may adjust the audio signal level of the separated target object audio signals using a preset mode or individually. For example, when the user activates a variable preset mode through an automatic adjustment interface (830), the audio signal processing device may adjust the magnitude of the audio signal level of each detected target object audio signal to a magnitude determined by Equation 1. The size determined by mathematical formula 1 can represent a size (level) determined by the ratio of the size of the target object audio signal to the size of the audio input signal.

[0113]

[0114] FIG. 9 is a diagram illustrating an example in which an audio signal processing device is used in an image according to one embodiment.

[0115] Referring to FIG. 9, an audio signal processing device (e.g., the audio signal processing device (101) of FIG. 1) may be used to edit an audio signal included in an image (910). The image (910) may be a still image, a frame of a video, or a video. When a user according to one embodiment activates an adaptive preset mode through an auto-adjustment interface (e.g., the auto-adjustment interface (830) of FIG. 8a), the audio signal processing device may adjust the audio signal level of the detected object audio signal (920) based on the association between the objects included in the image (910) and the detected object audio signal (920). For example, the image (910) may include a first person (911), a second person (912), and a piano (913), and the object audio signal (920) included in the image (910) may include a conversation sound (921), a music sound (922), and noise (923). Noise (923) may include the sound of a crowd murmuring, ambient noise occurring in a cafe, and train sounds. The object audio signal (920) may include, but is not limited to, the sound of conversation (921), music (922), noise (923), as well as wind sounds (mido).

[0116] The audio signal processing device can estimate the correlation between an object detected in an image (910) and an object audio signal (920) detected in an audio signal using a machine learning model (e.g., a transformer model or a multimodal basis model). For example, the audio signal processing device can estimate the correlation between a first person (911), a second person (912), a piano (913), and a conversation sound (921), a music sound (922), and a noise (923). The audio signal processing device can estimate that the first person (911), the second person (912), and the piano (913) have a high correlation with the conversation sound (921) and the music sound (922), and estimate that the first person (911), the second person (912), and the piano (913) have a low (or no) correlation with the noise (923). The audio signal processing device can increase the audio signal levels of the conversation sound (921) and music sound (922) estimated to be highly correlated, and decrease or mute the audio signal level of the noise (923) estimated to be low (or no) correlation. For example, the audio signal processing device can increase the audio signal level of the conversation sound (921) to 150% of the existing level, increase the audio signal level of the music sound (922) to 110% of the existing level, and decrease the audio signal level of the noise (920) to 30% of the existing level. Thus, the relative volumes of the target object audio signals can be adjusted based on the content of the image. For example, relative volume adjustment based on the content of the image can be useful in scenarios such as when a user captures an image while recording audio (e.g., recording a concert). Such scenarios can also occur when a user captures a video containing audio signals.

[0117] According to one embodiment, a user can individually adjust the levels of each object audio signal (920) for a conversation sound (921), a music sound (922), and a noise (923) (e.g., a siren sound). For example, the user can increase the audio signal level of the conversation sound (921) detected in the object audio signal (920) to 160% of the original level, the audio signal level of the music sound (922) to 130% of the original level, and the audio signal of the noise (923) to 0% of the original level. Adjusting the audio signal levels by the user can be performed by adjusting the weights of each detected target object audio signal. Since adjusting the audio signal levels through weight adjustment has been described in detail in FIG. 3, a redundant description will be omitted.

[0118] Since the audio editing device can be used to edit the object audio signal (920) included in the video (910) as described above, it can be used to edit the audio included in the content when a smartphone, tablet, TV (television; TV), PC (personal computer; PC) or VR (visual reality; VR) is used.

[0119]

[0120] FIG. 10 is a block diagram illustrating the configurations of an audio signal processing device according to one embodiment.

[0121] Referring to FIG. 10, the audio signal processing device (1000) may include a display (1010), a processor (1020), and a memory (1030). The audio signal processing device (1000) may correspond to the audio signal processing device (101) of FIG. 1.

[0122] The display (1010) can provide information visually. The display (1010) may correspond to or be included with the display module (160) of FIG. 1. The display (1010) may provide a user interface to provide information regarding the audio signal processing device. Additionally, the display (1010) may include a touch sensor configured to detect a touch and a pressure sensor that detects pressure caused by a touch in order to control the audio signal processing device.

[0123] Memory (1030) can store instructions that can be executed by the processor (1020). Memory (1030) may correspond to the memory (130) of FIG. 1. Instructions that can be executed by the processor (1020) may cause the processor (1020) to perform audio signal processing methods when executed by the processor (1020). Memory (1030) may be integrated with the processor (1020). For example, RAM (random access memory; RAM) or flash memory may be integrated with the processor (1020), such as an integrated circuit microprocessor. Memory (1030) may include separate devices, such as external disk drives, storage arrays, or other storage devices available by a database system. The memory (1030) and the processor (1020) can be operatively combined or communicate with each other through an I / O (input output; I / O) port, a network connection, etc., so that the processor (1020) can read a file stored in the memory (1030).

[0124] The processor (1020) can execute instructions stored in memory (1030). The processor (1020) may correspond to the processor (120) of FIG. 1. The processor (1020) may include a central processing unit (CPU), a graphics processing unit (GPU), a neural network processing unit (NPU), a media processing unit (MPU), a data processing unit (DPU), a vision processing unit (VPU), a video processor, an image processor, a display processor, a microprocessor, a processor core, a multi-core processor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or any combination thereof. When instructions are executed by the processor (1020), the processor (1020) can control the audio signal processing device (1000) to perform operations of the audio signal processing method described in this disclosure.

[0125] The audio signal processing device (1000) may include one or more processors, one or more memories for storing instructions, and a display for outputting an audio editing interface for editing an audio input signal. The audio signal processing device (1000) may generate an audio output signal by detecting one or more target object audio signals from the audio input signal using an object audio signal detection model that takes the audio input signal as input, providing an audio editing interface through the display in response to the detection of one or more target object audio signals, obtaining one or more target object audio signals separated from the audio input signal using an object audio signal separation model that takes the audio input signal as input, and adjusting the audio signal level of one or more separated target object audio signals in response to receiving an audio editing request input from a user through the audio editing interface.

[0126] The audio signal processing device (1000) determines that a target object audio signal has been detected from an audio input signal when the output value for a target object audio signal obtained from an object audio signal detection model is greater than a first threshold value, and the output value for the target object audio signal may correspond to a probability value of the existence of a target object audio signal in the audio input signal.

[0127] The audio signal processing device (1000) can provide section information indicating the section where one or more target object audio signals detected in the audio input signal are detected through an audio editing interface.

[0128] When an audio signal processing device (1000) receives an audio editing request input that corresponds to an automatic adjustment request to adjust the audio signal level of each of the separated target object audio signals from the audio input signal, the audio signal level of each of the separated target object audio signals can be adjusted to a level defined for each.

[0129] The audio signal processing device (1000) can determine that the target object audio signal is detected from the audio input signal when the magnitude value of the audio signal of the target object audio signal obtained from the sound source separation model is greater than the second threshold value.

[0130] The audio signal processing device (1000) can provide a waveform interface that displays the target object audio signal detected through the audio editing interface as a waveform.

[0131] When the audio signal processing device (1000) is an audio editing request input that is an automatic adjustment request to adjust the audio signal level of each of the target object audio signals separated from the audio input signal, the audio signal level of each of the separated target object audio signals can be adjusted to a level determined by the ratio of the size of the target object audio signal to the size of the audio input signal.

[0132]

[0133] An audio signal processing device (101, 1000) according to one embodiment comprises: one or more processors (120, 1020); and one or more memories (1030) for storing instructions; The audio signal processing device (101, 1000) may generate an audio output signal (203) by using an object audio signal detection model that takes the audio input signal (201) as input to detect one or more target object audio signals from the audio input signal (201), providing the audio editing interface through the display (1010) in response to the detection of the one or more target object audio signals, obtaining one or more target object audio signals separated from the audio input signal (201) using an object audio signal separation model that takes the audio input signal (201) as input, and adjusting the audio signal level of the one or more separated target object audio signals in response to receiving a user's audio editing request input (202) through the audio editing interface.

[0134] The object audio signal detection model above may be a classification-based model (510) that estimates whether the target object audio signal is detected in the audio input signal (201) based on sound event detection.

[0135] When the above instructions are executed individually or collectively by one or more processors (120, 1020), the audio signal processing device (101, 1000) determines that the target object audio signal has been detected from the audio input signal (201) when the output value for the target object audio signal obtained from the object audio signal detection model is greater than a first threshold value, and the output value for the target object audio signal may correspond to a probability value of the existence of the target object audio signal in the audio input signal (201).

[0136] When the above instructions are executed individually or collectively by the one or more processors (120, 1020), the audio signal processing device (101, 1000) may provide section information indicating the section in which the one or more detected target object audio signals are detected in the audio input signal (201) through the audio editing interface.

[0137] When the above instructions are executed individually or collectively by one or more processors (120, 1020), the audio signal processing device (101, 1000) can adjust the audio signal level of each of the separated target object audio signals to a level defined for each, if the audio editing request input (202) corresponds to an automatic adjustment request that adjusts the audio signal level of each of the separated target object audio signals from the audio input signal (201).

[0138] The object audio signal detection model may be a sound source separation model that takes the audio input signal (201) as input and outputs one or more target object audio signals presumed to be detected from the audio input signal (201).

[0139] The object audio signal detection model above may be a lightweight model of the object audio signal separation model above.

[0140] When the above instructions are executed individually or collectively by one or more processors (120, 1020), the audio signal processing device (101, 1000) may determine that the target object audio signal is detected from the audio input signal (201) if the magnitude value of the audio signal of the target object audio signal obtained from the sound source separation model is greater than a second threshold value.

[0141] When the above instructions are executed individually or collectively by one or more processors (120, 1020), the audio signal processing device (101, 1000) may provide a waveform interface (850) that represents the detected target object audio signal as a waveform through the audio editing interface.

[0142] When the above instructions are executed individually or collectively by one or more processors (120, 1020), the audio signal processing device (101, 1000) can adjust the audio signal level of each of the separated target object audio signals to a level determined by the ratio of the size of the target object audio signal to the size of the audio input signal (201), if the audio editing request input (202) is an automatic adjustment request for adjusting the audio signal level of each of the separated target object audio signals.

[0143] In the above audio signal processing device (101, 1000), each target object audio signal can be acquired in response to each detection of each target object audio signal.

[0144] In the above audio signal processing device (101, 1000), the audio editing interface may be provided before all of the one or more target object audio signals are separated from the audio input signal (201).

[0145] In the above audio signal processing device (101, 1000), the operation of acquiring one or more target object audio signals separated from the audio input signal (201) can be started before the operation of detecting the one or more audio signals from the audio input signals is completed.

[0146] In the above audio signal processing device (101, 1000), the audio input signal (201) includes a first target object audio signal and a second target object audio signal, and the acquisition of the first target object audio signal can be started in response to the detection of the first target object audio signal before the detection of the second target object audio signal.

[0147] In the above audio signal processing device (101, 1000), the operation of acquiring one or more target object audio signals separated from the audio input signal (201) can be performed only in the intervals of the audio input signal (201) where the one or more target object audio signals are detected.

[0148] An audio signal processing method performed by an audio signal processing device (101, 1000) according to one embodiment may include: an operation of detecting one or more target object audio signals from an audio input signal (201) using an object audio signal detection model that takes an audio input signal (201) as input; an operation of providing an audio editing interface through a display (1010) in response to the detection of the one or more target object audio signals; an operation of obtaining one or more target object audio signals separated from the audio input signal (201) using an object audio signal separation model that takes the audio input signal (201) as input; and an operation of generating an audio output signal (203) by adjusting the audio signal level of the one or more separated target object audio signals in response to receiving a user's audio editing request input (202) through the audio editing interface.

[0149] The audio signal processing method may further include an operation of determining that the target object audio signal is detected from the audio input signal (201) when the output value for the target object audio signal obtained from the object audio signal detection model is greater than a first threshold value. The output value for the target object audio signal may correspond to a probability value of the existence of the target object audio signal in the audio input signal (201).

[0150] The operation of providing the above audio editing interface may include providing section information indicating the section in which one or more detected target object audio signals are detected in the audio input signal (201) through the audio editing interface.

[0151] The operation of generating the above audio output signal (203) may include, when the audio editing request input (202) corresponds to an automatic adjustment request for adjusting the audio signal level of each of the separated target object audio signals from the audio input signal (201), adjusting the audio signal level of each of the separated target object audio signals to a level defined for each.

[0152] The audio signal processing method may further include an operation of determining that the target object audio signal is detected from the audio input signal (201) when the magnitude value of the audio signal of the target object audio signal obtained from the sound source separation model is greater than a second threshold value.

[0153] The operation of providing the above audio editing interface may include the operation of providing a waveform interface (850) that represents the detected target object audio signal as a waveform through the above audio editing interface.

[0154] The operation of generating the above audio output signal (203) may include, when the audio editing request input (202) is an automatic adjustment request for adjusting the audio signal level of each of the separated target object audio signals from the audio input signal (201), adjusting the audio signal level of each of the separated target object audio signals to a level determined by the ratio of the size of the target object audio signal to the size of the audio input signal (201).

[0155]

[0156] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination, and the program instructions recorded on the medium may be those specifically designed and configured for the embodiment or those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0157] The hardware device described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0158] Although the embodiments have been described above with reference to the limited drawings, those skilled in the art can apply various technical modifications and variations based thereon. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0159] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

1. In an audio signal processing device (101; 1000), One or more processors (120; 1020); One or more memories (130;1030) for storing instructions; and A display (1010) that outputs an audio editing interface for editing an audio input signal (201) Includes, When the above instructions are executed individually or collectively by one or more processors (120; 1020), the audio signal processing device (101; 1000) is, One or more target object audio signals are detected from the audio input signal (201) using an object audio signal detection model that takes the audio input signal (201) as input, and In response to the detection of one or more target object audio signals, the audio editing interface is provided through the display (1010), and One or more target object audio signals separated from the audio input signal (201) are obtained using an object audio signal separation model that takes the audio input signal (201) as input, and In response to receiving a user's audio editing request input (202) through the above audio editing interface, an audio output signal (203) is generated by adjusting the audio signal level of one or more separated target object audio signals. Audio signal processing device (101; 1000).

2. In Paragraph 1, The above object audio signal detection model is, A classification-based model (510) that estimates whether the target object audio signal is detected in the audio input signal (201) based on sound event detection, Audio signal processing device (101; 1000).

3. In Paragraph 1 or 2, When the above instructions are executed individually or collectively by one or more processors (120; 1020), the audio signal processing device (101; 1000) is, If the output value for the target object audio signal obtained from the object audio signal detection model is greater than the first threshold value, it is determined that the target object audio signal has been detected from the audio input signal (201). The output value for the target object audio signal is a probability value corresponding to the existence of the target object audio signal in the audio input signal (201). Audio signal processing device (101; 1000).

4. In any one of paragraphs 1 through 3, When the above instructions are executed individually or collectively by one or more processors (120; 1020), the audio signal processing device (101; 1000) is, Section information indicating a section of the audio input signal (201) in which one or more detected target object audio signals are detected in the audio input signal (201) is provided through the audio editing interface. Audio signal processing device (101; 1000).

5. In any one of paragraphs 1 through 4, When the above instructions are executed individually or collectively by one or more processors (120; 1020), the audio signal processing device (101; 1000) is, When the above audio editing request input (202) corresponds to an automatic adjustment request for adjusting the audio signal level of each of the target object audio signals separated from the audio input signal (201), the audio signal level of each of the separated target object audio signals is adjusted to a level defined for each. Audio signal processing device (101; 1000).

6. In Paragraph 1, The above object audio signal detection model is, A sound source separation model (710) that takes the above audio input signal (201) as input and outputs one or more target object audio signals presumed to be detected from the above audio input signal (201), Audio signal processing device (101; 1000).

7. In Paragraph 6, The above object audio signal detection model is, A lightweight model of the above object audio signal separation model, Audio signal processing device (101; 1000).

8. In Paragraph 6 or 7, When the above instructions are executed individually or collectively by one or more processors (120; 1020), the audio signal processing device (101; 1000) is, If the magnitude value of the audio signal of the target object audio signal obtained from the above sound source separation model (710) is greater than the second threshold value, it is determined that the target object audio signal has been detected from the audio input signal (201). Audio signal processing device (101; 1000).

9. In any one of paragraphs 6 through 8, When the above instructions are executed individually or collectively by one or more processors (120; 1020), the audio signal processing device (101; 1000) is, A waveform interface (850) representing the detected target object audio signal as a waveform is provided through the audio editing interface. Audio signal processing device (101; 1000).

10. In any one of paragraphs 6 through 9, When the above instructions are executed individually or collectively by one or more processors (120; 1020), the audio signal processing device (101; 1000) is, If the above audio editing request input (202) is an automatic adjustment request for adjusting the audio signal level of each of the target object audio signals separated from the audio input signal (201), the audio signal level of each of the separated target object audio signals is adjusted to a level determined by the ratio of the size of the target object audio signal to the size of the audio input signal (201). Audio signal processing device (101; 1000).

11. In any one of paragraphs 1 through 10, The above audio editing interface is provided before all of the one or more target object audio signals are separated from the audio input signal (201). Audio signal processing device (101; 1000).

12. In any one of paragraphs 1 through 11, The operation of acquiring one or more target object audio signals separated from the above audio input signal (201) starts before the operation of detecting the one or more audio signals from the above audio input signals is completed. Audio signal processing device (101; 1000).

13. In any one of paragraphs 1 through 12, The above audio input signal (201) includes a first target object audio signal and a second target object audio signal, and The acquisition of the first target object audio signal begins in response to the detection of the first target object audio signal before the detection of the second target object audio signal. Audio signal processing device (101; 1000).

14. In any one of paragraphs 1 through 13, The operation of acquiring one or more target object audio signals separated from the audio input signal (201) is performed only in the intervals of the audio input signal (201) where the one or more target object audio signals are detected. Audio signal processing device (101; 1000).

15. An audio signal processing method performed by an audio signal processing device (101; 1000), wherein An operation of detecting one or more target object audio signals from the audio input signal (201) using an object audio signal detection model that takes the audio input signal (201) as input; An operation of providing an audio editing interface through a display (1010) in response to the detection of one or more target object audio signals; An operation of obtaining one or more target object audio signals separated from the audio input signal (201) using an object audio signal separation model that takes the audio input signal (201) as input; and An operation to generate an audio output signal (203) by adjusting the audio signal level of one or more separated target object audio signals in response to receiving a user's audio editing request input (202) through the audio editing interface. including, Audio signal processing method.

Images