Attention calling method, device, and program

Abstract

One aspect of the disclosed technology comprises: an acquisition unit 1 that acquires acoustic signals emitted from a plurality of sound sources and arriving at a user; and a control unit 2 that, when the acoustic signals include an attention acoustic signal, performs control such that the user becomes more easily aware of the attention acoustic signal. The attention acoustic signal is an acoustic signal emitted from a sound source requiring attention. The control is performed using sound emitted from an open type acoustic sound emission device worn by the user.

Description

Attention calling method, device and program

[0001] The disclosed technology relates to a technology for calling a user's attention.

[0002] A technology using a characteristic sound to call a user's attention is known (for example, see Non-Patent Document 1). This enables more effective attention calling. An example of the characteristic sound is a warning sound.

[0003] Sonoko Kuwano, "Psychological Evaluation of Warning Signal Sounds", [online], 2001, [searched on April 2, 2025], Internet <URL: https: / / www.jstage.jst.go.jp / article / souonseigyo1977 / 25 / 1 / 25_1_3 / _pdf / -char / ja>

[0004] It is reasonable to call attention using a characteristic sound that unavoidably reaches the user's ears. However, such a characteristic sound may impose a psychological load on the user.

[0005] An objective of the disclosed technology is to provide an attention calling method, device and program that call a user's attention by a method different from conventional ones.

[0006] One aspect of the disclosed technology includes an acquisition unit that acquires acoustic signals emitted from a plurality of sound sources approaching a user, and a control unit that performs control to make the user easily notice an attention acoustic signal when the acoustic signals include the attention acoustic signal. The attention acoustic signal is an acoustic signal emitted from a sound source requiring attention. The control is performed using sound emitted from an open-type acoustic emission device worn by the user.

[0007] According to the disclosed technology, it is possible to call a user's attention by a method different from conventional ones.

[0008] Figure 1 is a diagram illustrating an example of the functional configuration of the attention calling device. Figure 2 is a diagram illustrating an example of the processing procedure of the attention calling method. Figure 3 is a diagram for explaining an experimental example. Figure 4 is a diagram illustrating an example of the functional configuration of a computer.

[0009] Embodiments of the disclosed technology will be described below with reference to the drawings. Note that components having the same function are numbered identically in the drawings, and redundant explanations are omitted.

[0010] [Warning device and method] The warning device, as shown in Figure 1, includes, for example, an acquisition unit 1 and a control unit 2. The control unit 2 includes, for example, a separation unit 21, a processing unit 22, and a presentation unit 23.

[0011] The warning method is implemented, for example, by having each component of the warning device perform the processing from step S1 to step S2 shown in Figure 2.

[0012] The user is assumed to be wearing an open-type acoustic sound emission device. An example of an open-type acoustic sound emission device is an open-type earphone with a microphone. By using an open-type acoustic sound emission device, it becomes possible to hear both processed sound (sound that has been acoustically processed) and natural sound. Alternatively, a closed-type acoustic sound emission device with a so-called hear-through function (a function that picks up incoming acoustic signals with a microphone and emits them from the driver unit) may be used instead of an open-type acoustic sound emission device. Also, "acoustic sound emission device" may be read as "sound emission device".

[0013] For example, suppose a user is walking in the city and a car is approaching from behind, requiring them to avoid it. The warning device and method would, in such a situation, acquire the sound of the approaching car with a microphone, process it into a sound that will alert the user, and present it through an open-type acoustic sound emitter. Note that "warning" includes the act of notifying, acknowledging, or presenting the warning content.

[0014] The following describes each component of the warning device.

[0015] <Acquisition Unit 1> The acquisition unit 1 acquires acoustic signals emitted from multiple sound sources arriving at the user (step S1). The acquired acoustic signals are output to the control unit 2.

[0016] The acquisition unit 1 acquires sounds from the user's surroundings, which are picked up by a microphone. In this case, the microphone may be one attached to an open-type acoustic sound emission device, or it may be an external microphone. The microphone may be positioned close to the user in order to acquire sounds from the user's surroundings.

[0017] Acoustic signals may include warning signals. Warning signals are acoustic signals emitted from sound sources that require attention. Examples of warning signals include hazardous sounds such as the sound of an approaching vehicle.

[0018] <Control Unit 2> The control unit 2 receives the acoustic signal acquired by the acquisition unit 1.

[0019] If the acoustic signal includes a warning acoustic signal, the control unit 2 controls the system to make it easier for the user to notice the warning acoustic signal (step S2).

[0020] Control by the control unit 2 is performed using sound emitted from an open-type acoustic sound emission device worn by the user.

[0021] The control by the control unit 2 may be performed by using the user as a reference and emphasizing the acoustic signals arriving at the user from the direction from which the warning acoustic signal originated.

[0022] The following describes an example of the processing performed by the control unit 2. As shown in Figure 1, the control unit 2 includes, for example, a separation unit 21, a processing unit 22, and a presentation unit 23.

[0023] The separation unit 21 extracts a warning sound signal from the input sound signal. The extracted warning sound signal is output to the processing unit 22. If there are multiple warning sound signals and multiple warning sound signals are extracted, the multiple warning sound signals are output to the processing unit 23 in the order in which the corresponding sound signals were acquired by the acquisition unit 1.

[0024] Existing techniques can be used to extract the attention acoustic signal. Examples of such existing techniques are described in references 1 and 2.

[0025] [Reference 1] Marc Delcroix (et al.), "AI that listens to the voices of those it wants to hear—SpeakerBeam, a selective speech listening technology based on deep learning," [online], 2021, [Retrieved April 3, 2025], Internet <URL: https: / / journal.ntt.co.jp / wp-content / uploads / 2021 / 07 / JN20210722.pdf>

[0026] [Reference 2] Nippon Telegraph and Telephone Corporation, "Devising a New Audio Signal Processing Method to Extract Only Interesting Information - ConceptBeam: A Filter that Separates and Extracts Speech by Meaning," [online], May 30, 2023, [Retrieved April 3, 2025], Internet <URL: https: / / group.ntt / jp / newsrelease / 2023 / 05 / 30 / 230530d.html>

[0027] For example, the separation unit 21 uses a predetermined sound source separation algorithm to extract sounds from the acoustic signal that require attention. Sounds that require attention are sounds that, when heard by a user, may require the user to take some action to avoid danger. Examples of such sounds include danger sounds, screams, collision sounds, and the sound of glass breaking. Examples of sound source separation algorithms include ICA, NMF, and DNN-based algorithms.

[0028] For example, if a feature database of attention acoustic signals as target sounds has been created in advance, the separation unit 21 will, for example, use that feature database to extract the attention acoustic signals.

[0029] The separation unit 21 may separate the input acoustic signal into a plurality of acoustic signals for each direction of arrival. The separation unit 21 may then determine whether each separated acoustic signal is the sound of a sound source requiring attention. In this case, the separation unit 21 may output to the processing unit 22 the acoustic signal determined to be the sound of a sound source requiring attention, along with the direction of arrival corresponding to that acoustic signal. When the processing unit 22 performs acoustic processing using the direction of arrival, this direction of arrival may be used, for example.

[0030] The processing unit 22 performs predetermined acoustic processing on the attention sound signal extracted by the separation unit 21. The attention sound signal after acoustic processing is output to the presentation unit 23. The processing unit 22 performs predetermined acoustic processing, for example, at the timing when the attention sound signal is input to the processing unit 22.

[0031] An example of a specified acoustic processing is a process that increases the volume (sound pressure). In other words, this process makes the mixed sound originating from the attention acoustic signal have a higher sound pressure when the direct sound and sound from the acoustic emission device arrive at the eardrum compared to when the processing is not performed. Here, if the volume of the original attention acoustic signal is Vi and a predetermined number greater than 1 is n, the volume of the attention acoustic signal after acoustic processing, Vo, can be expressed as follows: Vo = n * Vi

[0032] An example of the specified acoustic processing is a process that convolves an HRTF that emphasizes the direction of arrival of the attention acoustic signal into Vi (see, for example, reference 3). By outputting the convolved audio as Vo, the direction of arrival of the attention acoustic signal can be made more recognizable to the user.

[0033] [Reference 3] Hashimoto, Miyu (et al.), "Study of Acoustic Conditions for the Construction of a Voice Evacuation Guidance System Utilizing Preceding Sound Effects," [online], 2018, [Retrieved April 3, 2025], Internet <URL: https: / / www.jstage.jst.go.jp / article / jasj / 74 / 9 / 74_507 / _pdf / -char / ja>, Journal of the Acoustical Society of Japan, Vol. 74, No. 9 (2018), pp. 507-510.

[0034] The presentation unit 23 superimposes the attention sound signal after acoustic processing by the processing unit 22 with the sound signal acquired by the acquisition unit 1, and presents the superimposed signal to the user. The sound signal acquired by the acquisition unit 1 is the original sound signal, and may be the sound signal before acquisition by the acquisition unit 1, or the sound signal acquired by the acquisition unit 1. By superimposing the attention sound signal after acoustic processing onto the original sound signal, it becomes possible to naturally direct the user's attention to the sound source that requires attention. This has been made clear by experiments described later.

[0035] In this example, the user is wearing an open-type acoustic device. The presentation unit 23 may also be this open-type acoustic device. In this case, the speaker of the open-type acoustic device outputs a warning acoustic signal after acoustic processing by the processing unit 22. The warning acoustic signal after acoustic processing emitted from the speaker of the open-type acoustic device and the original acoustic signal that has passed through the open part of the open-type acoustic device are superimposed in the space between the open-type acoustic device and the user's ear, and the superimposed signal is presented to the user.

[0036] The above process allows for alerting users in a different way than before. Specifically, it becomes possible to alert users without presenting them with distinctive sounds such as warning tones. As a result, for example, users can avoid danger without experiencing discomfort or stress.

[0037] [First Modification] The predetermined acoustic processing performed by the processing unit 22 of the control unit 2 may be acoustic processing other than changing the magnitude of the sound pressure. The predetermined acoustic processing only needs to be able to draw attention to a sound source that requires attention. Examples of such acoustic processing include changing the pitch period and adding echo (reverberation).

[0038] Furthermore, the prescribed sound processing may be varied depending on the degree of caution required. For example, the prescribed sound processing may be varied depending on the type of danger, such as the approach of a vehicle or the arrival of hazardous materials.

[0039] For example, in a given acoustic processing, the rate of increase in sound pressure may be changed. In other words, the given acoustic processing may be a time-dependent change in sound pressure.

[0040] By emphasizing the time-dependent changes in sound pressure (first derivative components), it is possible to give the user the impression that the sound source is rapidly approaching.

[0041] By emphasizing the acceleration change in sound pressure (second derivative component), it is possible to give the user the sensation that the sound source is approaching at an accelerating rate.

[0042] In this way, by changing the sound pressure over time, it becomes possible to convey to the user, for example, the power and tension of an approaching sound source.

[0043] In addition, as the predetermined acoustic processing, at least one of the acoustic processes described above may be performed.

[0044] In other words, the control by the control unit 2 may be at least one of pitch period conversion of the alert acoustic signal, reverberation addition, and temporal variation of sound pressure.

[0045] [Second Modified Example] The separation unit 21 of the control unit 2 may specialize the target sound to be extracted to a specific sound source. In other words, the separation unit 21 of the control unit 2 may perform extraction of an alert acoustic signal emitted only from a specific sound source.

[0046] In this case, the presentation unit 23 of the alert device may further output, from an open-type acoustic sound output device, an audio signal indicating the direction in which the user should move, in addition to the alert acoustic signal that has undergone acoustic processing by the processing unit 22. This enables the user to clearly recognize the evacuation direction.

[0047] Note that the audio of the direction in which the user should move, which is output from the open-type acoustic sound output device, may be presented to the user in a suppressed state. For example, the volume of the audio of the direction in which the user should move, which is output from the open-type acoustic sound output device, may be lower than the volume of the alert acoustic signal that has undergone acoustic processing by the processing unit 22 and is output from the speaker of the open-type acoustic sound output device.

[0048] [Third Modified Example] In the third modified example, the alert device may include a sensor 3 indicated by a broken line in FIG. 1. Examples of the sensor 3 are a camera, an infrared sensor, and a distance sensor. Sensing information obtained by the sensor 3 is input to the control unit 2.

[0049] In this case, the separation unit 21 may perform extraction of the alert acoustic signal using the sensing information obtained by the sensor 3.

[0050] When the sensor 3 is a camera, for example, it acquires in real time an image of a sound source requiring attention. The sensing information, which is the image acquired in real time, is input to the separation unit 21.

[0051] The separation unit 21 can obtain an attention acoustic signal by inputting, for example, the acoustic signal and sensing information into a predetermined neural network.

[0052] This allows, for example, the system to work in conjunction with sensor 3, which is a surveillance camera placed throughout the city, to detect vehicles approaching the user from behind a wall, extract the vehicle's operating sounds, and present them to the user.

[0053] [Experimental Example] As shown in Figure 3, the two speakers L and R are placed 1.5 m apart from each other.

[0054] From the perspective of the user, subject U, a male voice is emitted from speaker L on the left, and a crowd noise is emitted from speaker R on the right.

[0055] Subject U, seated 3.5 meters away from the midpoint between two speakers L and R, listens to the audio from the perspective of how a male's speech sounds. In this case, the sound source requiring attention is the male's speech.

[0056] The experiment was conducted using each of the following patterns (1) through (3).

[0057] (1) Only two speakers are used. Participants are asked to listen to a man's voice coming from the two speakers.

[0058] (2) Two speakers + headphones that cover the ears. Male speech is emitted from the headphones. Distance delay is added to the male speech output from the headphones.

[0059] (3) Two speakers + open-ear earphones. The open-ear earphones emit a processed alert sound signal. Ideally, an HRTF should be used, but for simplicity, sound is emitted only from the left speaker (L). Distance delay is also added to the sound from the open-ear earphones.

[0060] Experiments showed that while clarity was best in case (2) above, i.e., when using headphones, the lack of localization gave a strong feeling that a man was speaking right next to the ear. The result showed that direction was naturally discernible in case (3) above, with speakers + open-ear earphones.

[0061] (1) When listening through only two speakers, it was clear that a man was speaking, but the listeners were unable to hear the details of what he was saying.

[0062] (2) With speakers and over-ear headphones, the clarity of male voices improved, but it sounded as if the man was speaking right next to my ear. However, there was almost no sound localization.

[0063] (3) In the case of speakers + open-ear earphones, there is a localization of the direction of the male voice, and attention is naturally drawn towards the left speaker.

[0064] [Variations] The specific configuration of the embodiments of the disclosed technology is not limited to the configuration described above. The specific configuration of the embodiments of the disclosed technology can be modified as appropriate, without departing from the spirit of the embodiments of the disclosed technology.

[0065] The various processes described in the embodiments of the disclosed technology may be performed not only in chronological order according to the order described, but also in parallel or individually as required by the processing capacity of the device performing the processes.

[0066] For example, data exchange between components of the alert device may occur directly, or it may occur via a storage unit (not shown).

[0067] Furthermore, the present invention may also include a device (terminal) for using the apparatus, system, or method of the present invention via a network (telecommunication line). The "device (terminal) for use" may be equipped with functions necessary to obtain the effects of implementing the apparatus, system, or method of the present invention (for example, control functions, decoding functions, restoration functions, input / output functions, etc.).

[0068] It goes without saying that further modifications can be made as appropriate without departing from the spirit of this invention.

[0069] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually described as being incorporated by reference.

[0070] [Programs, Recording Media] The functions realized by the components described herein may be implemented in a circuitry or processing circuitry, including a general-purpose processor, an application-specific processor, an integrated circuit, an ASIC (Application Specific Integrated Circuit), a CPU (a Central Processing Unit), conventional circuits, and / or a combination thereof, programmed to realize the functions described herein. A processor includes transistors and other circuits and is considered a circuitry or processing circuitry. A processor may be a programmed processor that executes a program stored in memory.

[0071] In this specification, circuitry, unit, and means are hardware programmed to perform or execute the functions described herein. Such hardware may be any hardware disclosed herein, or any hardware known to be programmed to perform or execute the functions described herein.

[0072] If the hardware is a processor that is considered to be a type of circuitry, then the circuitry, means, or unit is a combination of hardware and software used to constitute the hardware and / or processor.

[0073] The various processes described above can be carried out by loading a program that executes each step of the above method into the recording unit 2020 of the computer 2000 shown in Figure 4, and then causing the control unit 2010, input unit 2030, output unit 2040, display unit 2050, etc. to operate.

[0074] The program describing this process can be recorded on a computer-readable recording medium. Any computer-readable recording medium can be used, such as a magnetic recording device, optical disc, magneto-optical recording medium, or semiconductor memory.

[0075] A program describing this process may be included in a computer program product.

[0076] Furthermore, this program may be distributed, for example, by selling, transferring, or lending portable recording media such as DVDs or CD-ROMs on which the program is recorded. Alternatively, the program may be stored in the storage device of a server computer and distributed by transferring the program from the server computer to other computers via a network.

[0077] A computer executing such a program may, for example, first store the program recorded on a portable storage medium or a program transferred from a server computer in its own storage device. Then, when processing is to be executed, the computer reads the program stored on its own storage medium and executes the processing according to the read program. Alternatively, the computer may directly read the program from the portable storage medium and execute the processing according to that program, or it may sequentially execute the processing according to the received program each time a program is transferred to it from a server computer. Furthermore, the processing may be executed using a so-called ASP (Application Service Provider) type service, where the processing function is realized only by issuing execution instructions and obtaining results, without transferring the program from the server computer to this computer.In addition, the processing may be executed using a so-called SaaS (Software as a Service) type service, where a part of the server computer is made available to the user along with the program. Furthermore, the term "program" in this form includes information used for processing by an electronic computer that is equivalent to a program (data, etc., that is not a direct instruction to the computer but has the property of defining the processing of the computer).

[0078] Furthermore, in this configuration, the device is configured by executing a predetermined program on a computer, but at least a part of these processes may be implemented in hardware.

Claims

1. A method for alerting a user, comprising the steps of: an acquisition unit acquiring acoustic signals emitted from multiple sound sources arriving at the user; and a control unit controlling the acoustic signals to make the user more aware of the alert acoustic signals if the acoustic signals include an alert acoustic signal, wherein the alert acoustic signal is an acoustic signal emitted from a sound source requiring attention, and the control is performed using sound emitted from an open-type acoustic sound emitter worn by the user.

2. The warning method according to claim 1, wherein the control is based on the user and emphasizes the sound signal arriving to the user from the direction from which the warning sound signal originated.

3. The warning method according to claim 1, wherein the control is at least one of the conversion of the pitch period of the warning sound signal, the addition of reverberation, and the time change of sound pressure.

4. The warning method according to claims 1 to 3, further comprising emitting an audio signal from the open-type acoustic sound emission device indicating the direction in which the user should move.

5. An alerting device comprising: an acquisition unit that acquires acoustic signals emitted from multiple sound sources arriving at a user; and a control unit that, if the acoustic signals include an alert acoustic signal, controls the device to make the user more aware of the alert acoustic signal, wherein the alert acoustic signal is an acoustic signal emitted from a sound source requiring attention, and the control is performed using sound emitted from an open-type acoustic sound emitter worn by the user.

6. A program for causing a computer to perform each step of the attention-gathering method of claim 1.

Images