Task assistance system and portable device

Abstract

[Problem] To provide a task assistance system with which it is possible to record the behavior of a designated party who is engaged in a task and execute task assistance for the designated party. [Solution] A task assistance system monitors and manages the behavior of a designated party who is engaged in a task, and assists performance of the task by the designated party. The task assistance system comprises: a portable device worn by the designated party; an administrator terminal on which an image captured by the portable device is displayed, and which is capable of communicating with the portable device; and a task assistance server that configures and stores designated-party behavior information on the basis of data acquired by the portable device, and controls communication between the portable device and the administrator terminal. The portable device and the administrator terminal are connected to a task assistance server via a communication line. The portable device includes a headset part, a battery unit part, and a communication power cable that connects the headset part and the battery unit part.

Description

Business support systems and mobile devices

[0001] This invention relates to a business support system, administrator terminal, business support server, and portable device that monitor and manage the actions of personnel engaged in business and support the performance of those personnel.

[0002] Conventionally, the actions of workers have been recorded, and to date, the action recording system described in Patent Document 1 is known. This action recording system has a terminal device that is worn by the worker, and an imaging unit incorporated in this terminal device captures the surrounding situation, and the captured images are recorded.

[0003] Japanese Patent Publication No. 2018-037951

[0004] In recent years, due to a shortage of personnel, there has been an increase in situations where less experienced staff members are handling tasks independently, and there is a growing demand for real-time advice and support for such staff members. However, the activity recording system described in Patent Document 1 did not have a function to support the work of such staff members.

[0005] This invention has been made in view of these circumstances, and the object of this invention is to provide a business support system that can record the actions of an employee and provide business support to that employee. This business support system can be used for training to perform the business, securing and recording the knowledge (information) necessary to perform the business, protecting the employee performing the business and securing evidence, and improving the governance of the business.

[0006] A business support system according to an embodiment of one aspect of the present invention is a business support system that monitors and manages the actions of an employee engaged in business and supports the employee in performing business, comprising: a portable device worn by the employee; an administrator terminal that displays images captured by the portable device and allows communication with the portable device; and a business support server that constructs and stores employee action information based on data acquired by the portable device and controls communication between the portable device and the administrator terminal, wherein the portable device and the administrator terminal are connected to the business support server via a communication line, the portable device includes a headset unit, a battery unit, and a communication power cable connecting the headset unit and the battery unit, the headset unit includes: a camera unit that captures images of the employee's surroundings, a microphone unit that generates employee surroundings audio data when sound from the employee's surroundings is input, an earphone unit that generates audio to be transmitted to the employee based on the audio data, and an audio communication control unit for the employee to operate in order to control the operation of the microphone unit and the earphone unit and communicate audio data, the battery unit is The system includes: an image transmission unit that transmits images captured by the camera unit to the business support server via the communication line; an ambient sound transmission unit that transmits ambient sound of the person in charge acquired by the microphone unit to the business support server via the communication line; an operation input and video display unit that receives operation input from the person in charge and displays video; an ambient sound reception unit that receives ambient sound data transmitted from the business support server, which is ambient sound for the person in charge to be generated from the earphone unit; an ambient image reception unit that receives ambient image data transmitted from the business support server, which is ambient image for the person in charge to be displayed on the operation input and video display unit; and a battery unit that supplies power to the headset unit via the communication power cable for the headset unit to operate and for the battery unit to operate.The system includes a control unit that controls the operation of the mobile device, and the control unit controls the captured image transmission unit, the person in charge ambient sound transmission unit, the person in charge voice reception unit, the person in charge image reception unit, the microphone unit, the earphone unit, the camera unit, and the operation input / video display unit so that the mobile device and the administrator terminal transmit audio and video bidirectionally via the business support server and the communication line, thereby causing the person in charge image transmitted from the administrator terminal to be displayed on the operation input / video display unit and the captured image captured by the camera unit to be transmitted, and causing the person in charge voice transmitted from the administrator terminal to be generated from the earphone unit and the person in charge ambient sound acquired by the microphone unit to be transmitted.

[0007] In the aforementioned business support system, the image transmitted to the person in charge is characterized in that it includes video containing information necessary for the person in charge to perform the aforementioned tasks.

[0008] In the aforementioned business support system, the voice communication of the person in charge is characterized in that it includes voice indicating information necessary for the person in charge to perform the aforementioned business.

[0009] The aforementioned business support system is characterized in that video, photographic, and / or memo information relating to education and / or technical transfer for performing the aforementioned business is stored in the portable device storage unit of the battery unit of the portable device.

[0010] In the aforementioned business support system, the administrator terminal is characterized by comprising: a display screen receiving unit that receives an image display screen showing the captured image sent from the business support server; an administrator terminal display unit that displays the received image display screen; an output voice receiving unit that receives the surrounding voice of the person in charge sent from the business support server; a voice generating unit that generates the received surrounding voice of the person in charge; a transmission voice input unit that receives the voice transmitted by the person in charge; a transmission voice transmission unit that sends the input voice transmitted by the person in charge toward the business support server; an action information provision request unit that sends an action information provision request toward the business support server requesting the person in charge's action information; an action information receiving unit that receives the person in charge's action information provided from the business support server in response to the action information provision request; a transmission image input unit that receives or constitutes the person in charge's transmission image; and a transmission image transmission unit that sends the input or constituted person in charge's transmission image toward the business support server.

[0011] In the aforementioned business support system, the business support server includes: a captured image receiving unit that receives the captured images sent from the captured image sending unit; a person-in-charge surrounding voice receiving unit that receives the person-in-charge surrounding voice sending unit; an action information configuring unit that constitutes the person-in-charge action information including the received captured images and the person-in-charge surrounding voice; an action information storage unit that stores the configured person-in-charge action information; a display screen configuring unit that processes the captured images received by the captured image receiving unit to constitute the image display screen; a display screen providing unit that provides the configured image display screen to the administrator terminal; a transmission voice receiving unit that receives the person-in-charge transmission voice sent from the transmission voice sending unit; and a transmission image receiving unit that receives the person-in-charge transmission images sent from the transmission image sending unit K. The system includes: a call establishment unit that establishes an audio and video call between the mobile device and the administrator terminal by sending the surrounding voice of the person in charge received by the person in charge surrounding voice reception unit to the administrator terminal, sending the person in charge transmission voice received by the transmission voice reception unit to the mobile device, sending an image display screen including the captured image provided by the display screen provision unit to the administrator terminal, and sending the person in charge transmission image received by the transmission image reception unit to the mobile device; an action information request reception unit that receives the action information provision request sent from the action information provision request unit; and an action information provision unit that provides the person in charge action information read from the action information storage unit to the administrator terminal in response to the received action information provision request, wherein when the action information reception unit receives the person in charge action information, the received person in charge action information is displayed on the administrator terminal display unit.

[0012] In the aforementioned business support system, the earphone section is characterized by being composed of bone conduction earphones, canal-type earphones, or in-ear earphones.

[0013] The aforementioned business support system is characterized in that the camera unit, the microphone unit, and the earphone unit are integrated as a headset, and the headset is configured to be worn on the head of the person in charge.

[0014] In the aforementioned business support system, the portable device has a camera stop mode, in which the camera unit stops taking pictures and the microphone unit stops acquiring audio, and a notification sound is emitted from the earphone unit to indicate that the camera stop mode is active.

[0015] In the aforementioned business support system, the mobile device has a call mode, and in the call mode, it is possible to make calls between the mobile device and the administrator terminal, and the captured image taken by the camera unit is highlighted on the administrator terminal display unit.

[0016] In the aforementioned business support system, the mobile device includes a location information detection unit that receives radio waves from a satellite positioning system and calculates location information, and a location information transmission unit that transmits the calculated location information to the business support server, the business support server includes a location information reception unit that receives the location information transmitted from the location information transmission unit, a map information request unit that sends a map information provision request requesting map information for the area surrounding the received location information, and a map information reception unit that receives the map information provided in response to the map information provision request, the activity information configuration unit configures the person in charge activity information so as to link the location information received by the location information reception unit with the captured image received by the captured image reception unit and the person in charge's surrounding voice received by the person in charge's surrounding voice reception unit, and the display screen configuration unit creates a location information image that displays the location information received by the location information reception unit as the location of the mobile device on the map of the map information received by the map information reception unit. The image display screen is configured such that the captured image received by the captured image receiving unit and the location information image are displayed on the same screen.

[0017] In the aforementioned business support system, the business support server has a speech-to-text generation unit that generates text from speech, the speech-to-text generation unit generates the surrounding speech of the person in charge received by the person in charge surrounding speech reception unit as a surrounding speech string, and the action information configuration unit configures the person in charge action information such that it links the surrounding speech string generated by the speech-to-text generation unit to the captured image received by the captured image reception unit and the surrounding speech of the person in charge received by the person in charge surrounding speech reception unit.

[0018] In the aforementioned business support system, the business support server has a verbal abuse determination unit that determines whether verbal abuse is present, the verbal abuse determination unit determines whether the surrounding voice string generated by the voice-to-text generation unit contains abusive words that have been registered in advance, and generates verbal abuse information based on the determination result, and the behavior information configuration unit configures the person in charge behavior information to include the verbal abuse information generated by the verbal abuse determination unit.

[0019] In the aforementioned business support system, the business support server has a voice analysis unit that analyzes voice to estimate the mental state, the voice analysis unit analyzes the voice surrounding the person in charge received by the person in charge surrounding voice reception unit and the voice-to-text string generated by the voice-to-text generation unit to estimate the mental state of the person in charge, and generates person in charge mental state information based on the estimated mental state of the person in charge, and the behavior information configuration unit configures the person in charge behavior information to include the person in charge mental state information generated by the voice analysis unit.

[0020] In the aforementioned business support system, the business support server includes: a face recognition unit that authenticates the face of the person being cared for in the captured image received by the captured image reception unit and identifies the name of the person being cared for; an expression analysis unit that analyzes the expression of the person being cared for in the captured image received by the captured image reception unit, estimates at least one of emotions such as joy, anger, sadness, and happiness, energy level, and satisfaction level, and calculates an estimated expression value that quantifies the estimation result; a person being cared for information configuration unit that configures person being cared for information including the name of the person being cared for identified by the face recognition unit and the estimated expression value obtained by the expression analysis unit; a person being cared for information storage unit that stores the configured person being cared for information; a person being cared for information provision request reception unit that receives a request for person being cared for from the administrator terminal; and a person being cared for information provision unit that provides the person being cared for, read from the person being cared for information storage unit in response to the received request for person being cared for, to the administrator terminal. The administrator terminal is: The system comprises: a care recipient information request unit that sends the care recipient information request to the business support server; and a care recipient information receiving unit that receives the care recipient information provided by the business support server in response to the care recipient information request, wherein when the care recipient information receiving unit receives the care recipient information, the received care recipient information is displayed on the administrator terminal display unit.

[0021] In the aforementioned business support system, the business support server has a face-to-face time measurement unit that measures the face-to-face time during which the person to be cared for is visible in the captured image received by the captured image reception unit, and the person to be cared for information configuration unit is characterized in that it configures the person to be cared for so as to include the face-to-face time measured by the face-to-face time measurement unit.

[0022] In the aforementioned business support system, the business support server has an image and sound analysis unit that performs analysis based on the captured image and the surrounding voice of the person in charge, the image and sound analysis unit estimates the actions of the person in charge from the captured image received by the captured image receiving unit, extracts the statements of the person in charge from the surrounding voice of the person in charge received by the surrounding voice receiving unit, and determines the person in charge's work evaluation based on the person in charge's actions and statements, and the action information configuration unit configures the person in charge's action information to include the work evaluation determined by the image and sound analysis unit.

[0023] In the aforementioned business support system, the business support server has an image-sound analysis unit that performs analysis based on the captured image and the surrounding voice of the person in charge, the image-sound analysis unit estimates the actions of the person in charge from the captured image received by the captured image reception unit, extracts the statements of the person in charge from the surrounding voice of the person in charge received by the surrounding voice reception unit, and determines the performance evaluation of the person in charge based on the actions and statements of the person in charge, the action information configuration unit configures the person in charge action information to include the performance evaluation determined by the image-sound analysis unit, the image-sound analysis unit extracts statements questioning the person in care from the surrounding voice of the person in care received by the surrounding voice reception unit, estimates the person in care's response to the questioning statements from the actions of the person in care captured in the captured image received by the captured image reception unit, and determines the risk of hearing loss or dementia of the person in care, and the person in care information configuration unit configures the person in care information to include the risk of hearing loss or dementia of the person in care determined by the image-sound analysis unit.

[0024] The business support system is characterized in that it has a plurality of mobile devices, the business support server has an administrator absence determination unit that determines the absence of the administrator operating the administrator terminal, and when one of the plurality of mobile devices switches to the call mode, the administrator absence determination unit determines that the administrator is absent, making it possible to make a call between the plurality of mobile devices.

[0025] Applicable to a business support system that monitors and manages the actions of personnel engaged in business and supports the performance of business by said personnel, a portable device worn by said personnel, the business support system comprises an administrator terminal that displays images captured by the portable device and can communicate with the portable device, and a business support server that constructs and stores personnel action information based on data acquired by the portable device and controls communication between the portable device and the administrator terminal, the portable device and the administrator terminal are connected to the business support server via a communication line, the portable device includes a headset unit, a battery unit, and a communication power cable connecting the headset unit and the battery unit, the headset unit includes a camera unit that captures images of the personnel's surroundings, a microphone unit that generates personnel's surrounding voice data when sound from the personnel's surroundings is input, an earphone unit that generates voice to be transmitted to the personnel based on the voice data, and an audio communication control unit for the personnel to operate in order to control the operation of the microphone unit and the earphone unit and to communicate voice data, the battery unit is The portable device comprises: an image transmission unit that transmits images captured by the camera unit to the business support server via the communication line; an ambient sound transmission unit that transmits ambient sound of the person in charge acquired by the microphone unit to the business support server via the communication line; an operation input and video display unit that receives operation input from the person in charge and displays video; an ambient sound reception unit that receives ambient sound data transmitted from the business support server, which is ambient sound for the person in charge to be generated from the earphone unit; an ambient image reception unit that receives ambient image data transmitted from the business support server, which is ambient image for the person in charge to be displayed on the operation input and video display unit; a battery unit that supplies power to the headset unit to operate via the communication power cable, and also supplies power to the battery unit to operate; and a control unit that controls the operation of the portable device.The portable device is characterized in that, in order for the portable device and the administrator terminal to transmit audio and video bidirectionally via the business support server and the communication line, the control unit controls the captured image transmission unit, the person in charge ambient audio transmission unit, the person in charge voice reception unit, the person in charge image reception unit, the microphone unit, the earphone unit, the camera unit, and the operation input / video display unit, thereby causing the person in charge image transmitted from the administrator terminal to be displayed on the operation input / video display unit and the captured image captured by the camera unit to be transmitted, and causing the person in charge voice transmitted from the administrator terminal to be generated from the earphone unit and the person in charge ambient audio acquired by the microphone unit to be transmitted.

[0026] Figure 1 is a diagram showing an example of the configuration of a business support system according to the embodiment. Figure 2 is a diagram showing an example of the functional blocks of a portable device that constitutes the business support system shown in Figure 1. Figure 3A is a diagram showing an example of the three-dimensional appearance of the headset portion of the portable device according to the embodiment, viewed from the front right diagonal direction. Figure 3B is a diagram showing an example of the three-dimensional appearance of the headset portion of the portable device according to the embodiment, viewed from the front left diagonal direction. Figure 3C is a diagram showing an example of the three-dimensional appearance of the headset portion of the portable device according to the embodiment, viewed from the side. Figure 3D is a diagram showing an example of the three-dimensional appearance of the headset portion of the portable device according to the embodiment, viewed from above. Figure 4A is a diagram showing an example of the three-dimensional appearance of the battery unit portion of the portable device according to the embodiment, viewed from the right diagonal direction. Figure 4B is a diagram showing an example of the three-dimensional appearance of the battery unit portion of the portable device according to the embodiment, viewed from the left diagonal direction. Figure 5 is a diagram showing an example of the functional blocks of the administrator terminal that constitutes the business support system shown in Figure 1. Figure 6 is a diagram showing an example of the functional blocks of the business support server that constitutes the business support system shown in Figure 1. Figure 7A is a diagram showing an example of the configuration of the person in charge action information according to the embodiment. Figure 7B is a diagram showing an example of the display of the person in charge mental state information included in the person in charge action information according to the embodiment. Figure 8 is a schematic diagram showing an example of the configuration of the image display screen shown on the administrator terminal according to the embodiment. Figure 9 is a diagram showing an example of the configuration of care recipient information according to the embodiment. Figure 10 is a diagram illustrating an example of the operation flow of the work support system performed during the work of the person in charge in the embodiment. Figure 11 is a diagram illustrating an example of the flow of analysis performed based on the configured person in charge behavior information in the work support system according to the embodiment.

[0027] [Embodiment] The embodiment of this invention will be described with reference to FIGS. 1 to 11. In the following embodiments, by way of example only, the business to which the business support system is applied will be described taking the care business as an example. However, the business to which the business support system according to the present invention is applied is not limited to the care business. That is, the business support system according to the present invention can be applied to other businesses such as, for example, construction businesses, childcare / education businesses, security businesses, maintenance businesses, administrative businesses such as police, emergency and fire departments, trading companies, manufacturing businesses, and medical businesses.

[0028] [Configuration of Business Support System] FIG. 1 is a diagram showing an example of the configuration of a business support system according to an embodiment. As described above, in this embodiment, the care business will be described as an example of the business. As forms of care, both in-facility care and home visit care are included. Also, caregivers (workers) engaged in caregiving work are referred to as responsible persons.

[0029] This business support system 1 includes a portable device 6 worn by a responsible person １ , 6 ２ ,..., 6 n and an administrator terminal 8 on which images taken by the portable device 6 １ , 6 ２ ,..., 6 ｎ are displayed and which is capable of making calls with the portable device 6 １ , 6 ２ ,..., 6n, and a business support server 2 that constructs and stores the responsible person behavior information 200 described later based on the data acquired by the portable device 6 １ , 6 ２ ,..., 6 n and controls the communication (information flow) between the portable device 6 １ , 6 ２ ,..., 6 n and the administrator terminal 8, and a map information providing server 10 that provides map information. These portable devices 6 １ , 6 ２ ,..., 6 nThe manager terminal 8, the business support server 2, and the map information providing server 10 are connected via the communication line 11, and can transmit and receive information to and from each other. The communication line 11 uses the Internet or the like. This business support system 1 monitors and manages the actions of the persons in charge of the business by recording the actions of the persons in charge of the business and performing business support for the persons in charge, and supports the execution of the business by the persons in charge outside the city.

[0030] In the following description, when it is not necessary to explicitly indicate that there are multiple, the mobile devices 6 １ 、6 ２ 、··、6 n are represented by the symbol of the mobile device 6 for the sake of simplicity.

[0031] [Mobile Device] FIG. 2 is a schematic functional block diagram of the mobile device 6. This mobile device 6 includes a control unit 60, a camera unit 61, a captured image transmission unit 62, a microphone unit 63, a surrounding voice transmission unit 64 for the person in charge, a transmitted voice reception unit 65 for the person in charge, an earphone unit 66, a position information detection unit 70, a position information transmission unit 71, an operation input and video display unit T, a battery unit E, a voice communication control operation unit G, a power button B1, operation buttons B2, B3, a transmitted image reception unit H for the person in charge, a mobile device storage unit 73, and a communication unit 74.

[0032] For example, in the present embodiment, the camera unit 61, the microphone unit 63, the earphone unit 66, and the voice communication control operation unit G are mounted on the headset unit 600 of the mobile device 6 shown in FIGS. 3A to 3D as described later.

[0033] Also, in the present embodiment, the control unit 60, the captured image transmission unit 62, the surrounding voice transmission unit 64 for the person in charge, the transmitted voice reception unit 65 for the person in charge, the position information detection unit 70, the position information transmission unit 71, the operation input and video display unit T, the battery unit E, the voice communication control operation unit G, the power button B1, the operation buttons B2, B3, the transmitted image reception unit H for the person in charge, the mobile device storage unit 73, and the communication unit 74 are mounted on the battery unit 620Z of the mobile device 6 shown in FIGS. 4A to 4B as described later.

[0034] The control unit 60 is configured to control the operations of each element of the mobile device 6. For example, this control unit 60 is configured to include a CPU (Central Processing Unit) not shown in the figure, a flash memory which is a non-volatile memory, and a RAM (Random Access Memory) which is a volatile memory. The CPU executes program execution, arithmetic processing, control of each element constituting the mobile device 6, etc. The flash memory stores programs executed by the CPU, registered data, etc., and the RAM is used as a work area for program execution and arithmetic processing by the CPU.

[0035] The camera unit 61 is configured to capture the surroundings of the person in charge and acquire a captured image. As the image sensor of this camera unit 61, a CMOS (Complementary Metal Oxide Semiconductor) area sensor, a CCD (Charged Coupled Device) area sensor, or the like can be used. The captured image transmission unit 62 transmits (sends) the captured image 802 captured by the camera unit 61 as image data to the business support server 2 via the communication line 11. The captured image 802 often constitutes a moving image, but may also be a still image.

[0036] The microphone unit 63 acquires the sound around the person in charge. That is, this microphone unit 63 is configured to generate the ambient sound of the person in charge, which is voice data, when the sound around the person in charge is input. As a specific operation, this microphone unit 63 senses the sound and converts it into an electrical signal.

[0037] The ambient sound transmission unit 64 of the person in charge is configured to transmit the ambient sound 208 of the person in charge acquired by the microphone unit 63 as voice data to the business support server 2 via the communication line 11.

[0038] The earphone unit 66 generates audio to be transmitted to the person in charge based on audio data. The person in charge transmission audio receiving unit 65 receives the person in charge transmission audio as audio data to be generated from the earphone unit 66. This person in charge transmission audio is sent from the business support server 2 as audio data via the communication line 11. The earphone unit 66 generates the audio to be transmitted to the person in charge. Specifically, the earphone unit 66 converts an electrical signal containing audio information (audio data) into sound waves.

[0039] The location information detection unit 70 receives radio waves from a satellite positioning system and calculates location information 212. A GPS (Global Positioning System) module can be used in this location information detection unit 70. The GPS module receives signals from the GPS system and notifies the current location on Earth. By obtaining location information 212 from the GPS module, the current location of the mobile device 6 can be obtained. The location information transmission unit 71 transmits the location information 212 calculated by the location information detection unit 70 as data to the business support server 2 via the communication line 11.

[0040] The Person in Charge Communication Image Reception Unit H is configured to receive person in charge communication images, which are image data sent from the business support server 2 via the communication line 11, for display on the operation input and video display unit T.

[0041] Furthermore, the operation input and video display unit T accepts operation input from the operator and also displays video.

[0042] The portable device storage unit 73 is composed of non-volatile memory. For example, an SD memory card can be used. The portable device storage unit 73 stores captured images 802, voice recordings of the person in charge 208, location information 212, etc. In the event that the connection to the communication line 11 is disconnected, the captured images 802 etc. can be temporarily stored in the portable device storage unit 73, and after the connection to the communication line 11 is restored, the temporarily stored captured images 802 etc. can be sent to the business support server 2.

[0043] The communication unit 74 of the battery unit 620Z performs data transmission and reception with the outside. This communication unit 74 has the function of connecting to the communication line 11 via Wi-Fi (registered trademark) communication and the function of connecting to the communication line 11 via a mobile communication line via LTE (Long Term Evolution) communication. When Wi-Fi communication is available, it connects to the communication line 11 via Wi-Fi communication, and when Wi-Fi communication is unavailable, it is controlled to connect to the communication line 11 via LTE communication.

[0044] The battery unit E supplies power to the headset unit 600 via the communication power cable 650, enabling the headset unit 600 to operate, and also supplies power to the battery unit 620Z to operate.

[0045] The portable device 6 includes a headset unit 600 (see Figures 3A to 3D), a battery unit unit 620Z (see Figures 4A to 4B), and a communication power cable 650 (see Figures 3C to 3D) connecting the headset unit 600 and the battery unit unit 620Z. The portable device 6 is configured such that the headset unit 600, which incorporates and integrates a camera unit 61, a microphone unit 63, and an earphone unit 66, and the battery unit unit 620Z, which includes a battery unit E that provides power and a communication unit 74 that has communication functions, are connected by the communication power cable 650.

[0046] Here, Figure 3A shows an example of the three-dimensional appearance of the headset portion of the portable device according to the embodiment, viewed from the front right diagonal direction. Figure 3B shows an example of the three-dimensional appearance of the headset portion of the portable device according to the embodiment, viewed from the front left diagonal direction. Figure 3C shows an example of the three-dimensional appearance of the headset portion of the portable device according to the embodiment, viewed from the side. Figure 3D shows an example of the three-dimensional appearance of the headset portion of the portable device according to the embodiment, viewed from above. Note that in Figures 3A and 3B, the communication power cable 650 is omitted from the notation and is not shown, but in Figures 3C and 3D, the communication power cable 650 is shown.

[0047] The headset portion 600 shown in Figures 3A to 3D is worn on the head of the person in charge (not shown). As shown in Figures 3A to 3D, a thin, wire-shaped connecting portion 612 made of an elastic material is attached along the back of the headset portion 600, and at both ends of this connecting portion 612, widened portions 614, which are roughly rectangular in side view and wider in the vertical direction, are formed. An electrical circuit board on which electronic components are mounted is embedded inside this widened portion 614.

[0048] Furthermore, for example, a video recording button and a still image capture button may be displayed on the operation input and video display unit T of the battery unit 620Z shown in Figures 4A and 4B, or operation buttons B2 and B3 may function as a video recording button and a still image capture button. The video recording button and the still image capture button may be located on, for example, the widening section 614 of the headset unit 600. For example, when the operator operates the video recording button, the camera unit 61 records a video, the recorded video is stored in the portable device storage unit 73, and the video data is sent to the business support server 2 via the communication line 11 and stored. Similarly, when the operator operates the still image capture button, the camera unit 61 captures a still image, the captured still image is stored in the portable device storage unit 73, and the image data is sent to the business support server 2 via the communication line 11 and stored.

[0049] Furthermore, for example, volume buttons may be displayed on the operation input and video display unit T of the battery unit 620Z shown in Figures 4A and 4B, or operation buttons B2 and B3 may function as volume buttons. These volume buttons may also be located on, for example, the widened portion 614 of the headset unit 600. When, for example, the operator operates these volume buttons, the volume generated from the earphone portion 66 of the headset unit 600 is adjusted.

[0050] Furthermore, as shown in Figures 3C to 3D, for example, a communication power cable connection terminal N1 is provided on the back side of the widened portion 614 of the headset portion 600, and a communication power cable 650 is inserted into this communication power cable connection terminal N1 to connect to the battery unit portion 620Z.

[0051] Furthermore, as shown in Figures 3A to 3D, for example, a leading portion 619 is formed at the tip of each ear hook portion 616, which houses a microphone portion 63 and an earphone portion 66. In this embodiment, a bone conduction earphone is used for the earphone portion 66. However, the earphone portion 66 is not limited to a bone conduction earphone; it may be another type of earphone, such as a canal-type earphone or an in-ear earphone.

[0052] Furthermore, as shown in Figures 3A to 3D, for example, a camera unit 61 is provided on the outer side of the left-hand front portion 619 so as to be rotatable in the vertical direction. The vertical rotation angle of the camera unit 61 is fixed at multiple predetermined positions. In addition, when performing caregiving tasks such as bathing or dressing, the camera unit 61 can be pointed upward to avoid photographing the person being cared for.

[0053] Furthermore, as shown in Figure 3B, for example, an audio communication control operation unit (operation button) G is provided on the outer side of the right-hand front end 619 of the headset unit 600. This audio communication control operation unit G is a button operated by the person in charge to control the operation of the microphone unit 63 and the earphone unit 66 in order to communicate voice data. When this audio communication control operation unit G is operated (pressed) by the person in charge, the mobile device 6 switches to call mode, and for example, it becomes possible to make a call between the mobile device 6 and the administrator terminal 8 (or another mobile device 6).

[0054] For example, by pressing down the voice communication control unit G once, the person in charge initiates voice communication between the mobile device 6 and the administrator terminal 8 as a function of the so-called Inter Communication System. Then, for example, by pressing down the voice communication control unit G again, the voice communication ends. Alternatively, for example, by pressing and holding the voice communication control unit G, voice communication between the mobile device 6 and the administrator terminal 8 may be performed for the duration of the press and hold.

[0055] Furthermore, for example, when the voice communication control unit G is operated and the system switches to call mode, the captured image 206 taken by the camera unit 61 is highlighted on the administrator terminal 8. In addition, video (images) transmitted from the administrator terminal 8 to the mobile device 6 may be displayed on the operation input and video display unit T of the battery unit 620Z.

[0056] Furthermore, as shown in Figure 4A, for example, a power button B1 is provided on the outer side of the battery unit 620Z. For example, by operating this power button B1 (for example, by pressing and holding it), the power can be switched on and off. Also, for example, by operating the power button B1 of the battery unit 620Z (by pressing it down for a short time), the camera can be switched between camera stop mode and camera operation mode. Note that the power button B1 may also be located on, for example, the widened portion 614 of the headset 600.

[0057] The mobile device 6 has a camera stop mode and a camera operation mode. In camera stop mode, the camera unit 61 stops taking pictures and the microphone unit 63 stops acquiring audio, and a notification sound is emitted from the earphone unit 66 to indicate that the camera is in stop mode. On the other hand, in camera operation mode, the camera unit 61 takes pictures and the microphone unit 63 acquires audio, and the notification sound indicating that the camera is in stop mode is stopped.

[0058] Furthermore, even when the headset unit 600 is connected to a commercially available PC (Personal Computer) or smartphone via a communication power cable 650, the camera unit 61, microphone unit 63, and earphone unit 66 will operate, allowing images and sounds to be sent to the PC or smartphone as image data and audio data via the communication line 11, or audio sent from the PC or other device as audio data via the communication line 11 to be listened to through the earphone unit 66.

[0059] Furthermore, in the configuration of the headset unit 600 shown in Figure 3, the microphone unit 63 is incorporated into the front end 619, but the configuration is not limited to this; it may also be provided at the tip of the camera unit 61. With this arrangement, the microphone unit 63 is positioned in front of the operator's mouth, making it easier to detect their voice.

[0060] Here, Figure 4A shows an example of the three-dimensional external view of the battery unit of the portable device according to the embodiment, viewed from the right at an oblique angle. Figure 4B also shows an example of the three-dimensional external view of the battery unit of the portable device according to the embodiment, viewed from the left at an oblique angle. This battery unit 620Z incorporates the previously described battery unit E, captured image transmission unit 62, person-in charge ambient voice transmission unit 64, person-in charge communication voice reception unit 65, location information detection unit 70, location information transmission unit 71, and communication unit 74, etc.

[0061] Furthermore, the battery unit 620Z can download and use various applications. This allows the portable device 6 to work in conjunction with various applications, such as nurse call applications and web conferencing applications, to improve efficiency in a way that suits each workplace where the work is performed.

[0062] As shown in Figures 4A and 4B, a communication power cable connection terminal N2 is provided at the lower end of the battery unit 620Z. By inserting the other end of the communication power cable 650 shown in Figures 3C and 3D into this communication power cable connection terminal N2, the headset unit 600 and the battery unit 620Z can be connected by the communication power cable 650.

[0063] Furthermore, an operation input and video display unit T is provided on the surface of the battery unit 620Z. As previously described, the operation input and video display unit T accepts operation input from the operator and also displays video. This operation input and video display unit T may also be configured to display the operating status of the portable device 6.

[0064] This operation input and video display unit T is composed of a display device, and for example, a display device such as a touch panel LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) is used.

[0065] The operation input and video display unit T of the battery unit 620Z may, for example, display Wi-Fi communication, LTE, and GPS information. This allows the system to indicate that these functions are operating.

[0066] Here, for example, so that the mobile device 6 and the administrator terminal 8 transmit audio and video bidirectionally via the business support server 2 and the communication line 11, the control unit 60 of the mobile device 6 controls the captured image transmission unit 62, the person in charge's surrounding audio transmission unit 64, the person in charge's transmitted audio reception unit 65, the person in charge's transmitted image reception unit H, the microphone unit 63, the earphone unit 66, the camera unit 61, and the operation input / video display unit T. As a result, the control unit 60 displays the person in charge's transmitted image sent from the administrator terminal 8 on the operation input / video display unit T and transmits the captured image taken by the camera unit 61, while also generating the person in charge's transmitted audio sent from the administrator terminal 8 through the earphone unit 66 and transmitting the person in charge's surrounding audio acquired by the microphone unit 63.

[0067] Furthermore, a locking mechanism (not shown) may be provided on the back of the battery unit 620Z so that it can be attached to the clothing of the person in charge. As previously described, the power for the portable device 6 is supplied from the battery unit E built into the battery unit 620Z via the communication power cable 650.

[0068] [Administrator Terminal] Figure 5 is a schematic diagram of the functional block of the administrator terminal 8. This administrator terminal 8 is configured to include a control unit 80, a display screen receiving unit 81, an administrator terminal display unit 82, an output voice receiving unit 83, a voice generation unit 84, a transmitted voice input unit 85, a transmitted voice transmission unit 86, a transmitted image input unit J, a transmitted image transmission unit K, an action information provision request unit 90, an action information receiving unit 91, an administrator terminal storage unit 93, a care recipient information provision request unit 94, a care recipient information receiving unit 95, an input unit 98, and a communication unit 99. An information terminal such as a PC, smartphone, or tablet can be used as the administrator terminal 8.

[0069] The control unit 80 is configured to include a CPU (not shown), an auxiliary storage device which is a non-volatile memory device, and RAM. The CPU performs tasks such as executing programs and controlling each element that makes up the administrator terminal 8. The auxiliary storage device stores programs executed by the CPU, and uses HDDs (Hard Disk Drives) or SSDs (Solid State Drives).

[0070] The display screen receiving unit 81 receives an image display screen 800 on which the captured image 802 is displayed as image data. The image display screen 800 is sent from the business support server 2 via the communication line 11 as image data. The administrator terminal display unit 82 displays the image display screen 800 received by the display screen receiving unit 81. The administrator terminal display unit 82 is composed of a display device, and display devices such as a touch panel LCD (Liquid Crystal Display) or organic EL (Electro Luminescence) are used.

[0071] The output voice reception unit 83 receives the voice 208 of the person in charge, which is sent as voice data from the business support server 2 via the communication line 11. The voice generation unit 84 generates the voice 208 of the person in charge that was received by the output voice reception unit 83. A speaker or headphones can be used for the voice generation unit 84.

[0072] The voice transmission input unit 85 receives the voice transmission from the person in charge to be transmitted to the mobile device 6 by the administrator operating the administrator terminal 8. A microphone is used in the voice transmission input unit 85. The voice transmission output unit 86 transmits the voice transmission from the person in charge input to the voice transmission input unit 85 as voice data to the business support server 2 via the communication line 11. The voice transmission from the person in charge may include voices indicating information necessary for the person in charge to perform the task (such as voices from an administrator explaining how to perform the task or voices explaining past cases).

[0073] The image transmission input unit J is configured to input or compose an image of the person in charge in response to the administrator's operation input to the administrator terminal 8. The image transmission input unit J may include a camera unit (not shown). In this case, the image of the person in charge may include video footage captured by the camera unit. The image of the person in charge may also include video footage containing information necessary for the person in charge to perform the task (such as images or videos showing how to perform the task, images or videos showing past cases, or video footage of the administrator operating the administrator terminal 8).

[0074] The image transmission unit K then transmits the image of the person in charge, which has been input or configured in the image input unit J, to the business support server 2.

[0075] The Behavioral Information Request Unit 90 sends a Behavioral Information Request as data to the Business Support Server 2 via the Communication Line 11, requesting the Behavioral Information of the Person in Charge 200. This Behavioral Information Request is executed by specifying the target of the request. For example, the Behavioral Information Request Unit 90 may specify the Person in Charge ID 202 and Person in Charge Name 204, or a combination of Person in Charge ID 202 and acquisition time 210, or a combination of Person in Charge ID 202, acquisition time 210, and the requested item being a captured image 206. Note that the acquisition time 210 specified by the Behavioral Information Request Unit 90 may be a specific time, or a start time and end time may be specified. The Behavioral Information of the Person in Charge 200 includes the captured image 206 acquired by the mobile device 6 and the surrounding voice 208 of the person in charge. The Behavioral Information Receiving Unit 91 receives the Behavioral Information of the Person in Charge 200 provided from the Business Support Server 2 in response to the Behavioral Information Request. When the action information receiving unit 91 receives the person in charge's action information 200, the received person in charge's action information 200 is displayed on the administrator terminal display unit 82. Specifically, the captured image 206, the surrounding voice string 214 generated from the person in charge's surrounding voice 208 included in the person in charge's action information 200, and other components included in the person in charge's action information 200 are displayed on the administrator terminal display unit 82. In addition, the person in charge's surrounding voice 208 included in the person in charge's action information 200 is generated from the voice generation unit 84.

[0076] The administrator terminal storage unit 93 uses a non-volatile storage device such as an HDD or SSD. This administrator terminal storage unit 93 stores the person in charge's action information 200, etc., received by the action information reception unit 91. The administrator terminal storage unit 93 also stores electronic files such as care records and care education and training manuals, which are created using the captured images 206 and surrounding voice 208 of the person in charge included in the person in charge's action information 200. For example, in response to the administrator's operation input, the transmission image input unit J of the administrator terminal 8 may configure the transmission image of the person in charge as an image containing the information necessary for the person in charge to perform the task, based on the electronic files such as care records and care education and training manuals.

[0077] The care recipient information request unit 94 sends the care recipient information request as image data to the business support server 2 via the communication line 11. This care recipient information request is executed by specifying the target of the request. For example, the care recipient ID 302 and the care recipient's name 304 may be specified, or a combination such as the care recipient ID 302 and the requested item being face-to-face time 308 may be specified.

[0078] The care recipient information receiving unit 95 receives care recipient information 300 provided from the business support server 2 in response to a request for care recipient information. When the care recipient information receiving unit 95 receives the care recipient information 300, the received care recipient information 300 is displayed on the administrator terminal display unit 82.

[0079] The input unit 98 consists of input devices for performing input, such as a keyboard, mouse, or touch panel. The communication unit 99 performs data transmission and reception with the outside world.

[0080] [Business Support Server] Figure 6 is a schematic diagram of the functional blocks of the business support server 2. This business support server 2 is configured to include a control unit 20, a captured image receiving unit 21, a staff member's surrounding voice receiving unit 22, an action information configuration unit 23, an action information storage unit 24, a display screen configuration unit 25, a display screen provision unit 26, a transmitted voice receiving unit 27, a call establishment unit 30, an administrator absence determination unit 31, an action information request receiving unit 32, an action information provision unit 33, a location information receiving unit 34, a map information request unit 35, a map information receiving unit 36, a facility drawing storage unit 37, a voice-to-text generation unit 38, an abusive language determination unit 40, a voice analysis unit 41, a face recognition unit 42, a facial expression analysis unit 43, a care recipient information configuration unit 44, a care recipient information storage unit 45, a care recipient information provision request receiving unit 46, a care recipient information provision unit 47, a face-to-face time measurement unit 50, an image-to-voice analysis unit 51, a communication unit 52, and a transmitted image receiving unit L. Business support server 2 uses server devices, PCs, etc.

[0081] The control unit 20 is configured to include a CPU, a non-volatile memory device, and RAM (not shown). The CPU performs tasks such as executing programs and controlling each element that makes up the business support server 2.

[0082] The captured image receiving unit 21 receives the captured image 802 transmitted as image data via the communication line 11 from the captured image transmission unit 62 of the mobile device 6. The person in charge's surrounding voice receiving unit 22 receives the person in charge's surrounding voice 208 transmitted as voice data via the communication line 11 from the person in charge's surrounding voice transmission unit 64 of the mobile device 6.

[0083] The behavior information structuring unit 23 is configured to construct employee behavior information 200 which includes the captured image 206 received by the captured image receiving unit 21 and the employee's surrounding voice 208 received by the employee's surrounding voice receiving unit 22.

[0084] The employee action information 200 is a collection of information that aggregates records of the employee's actions. By specifying extraction conditions, records of that employee's actions can be retrieved from the employee action information 200.

[0085] The behavior information structuring unit 23 divides the captured images 206 and the voice recordings of the person in charge 208 into predetermined time intervals, such as 5 minutes or 10 minutes, and associates the acquisition times 210 of the captured images 206, etc., with the person in charge's behavior information 200. By structuring the person in charge's behavior information 200 in this way, the person in charge's behavior information 200 can be retrieved from the person in charge's behavior information 200 by specifying the acquisition time 210.

[0086] Here, Figure 7A is a diagram showing an example of the configuration of employee action information according to the embodiment. Figure 7B is a diagram showing an example of the display of employee mental state information included in employee action information according to the embodiment. For example, as shown in Figure 7A, the employee ID (identification) 202 and employee name 204 included in the employee action information 200 are entered when logging in to the business support server 2 from the mobile device 6, such as when starting work. Alternatively, the employee ID 202 and employee name 204 are registered in the business support server 2 in advance when the mobile device 6 is assigned to an employee. The business support server 2 stores the employee ID 202 and employee name 204 associated with the mobile device ID that uniquely identifies the mobile device 6. When a captured image 206, etc. is transmitted from the mobile device 6 along with the mobile device ID, the business support server 2 identifies the employee ID 202 and employee name 204 from the received mobile device ID and configures the employee action information 200 so that the employee ID 202, etc. and the received captured image 206, etc. are stored. By referring to the person in charge ID 202, etc., it is possible to identify which person in charge the person in charge activity information 200 corresponds to. For example, by specifying the person in charge ID 202 and the person in charge name 204, it is possible to extract the person in charge activity information 200 corresponding to the specified person in charge, or by specifying a combination of person in charge ID 202 and acquisition time 210, it is possible to extract the captured image 206, the person in charge's surrounding audio 208, etc. from the person in charge activity information 200 that corresponds to that combination. The components of the person in charge activity information 200 shown in Figure 7A are just an example, and other elements may be included, or some of these elements may be deleted.

[0087] The action information storage unit 24 shown in Figure 6 stores the person in charge action information 200, which is configured in the action information configuration unit 23.

[0088] The display screen configuration unit 25 processes the captured image 206 received by the captured image reception unit 21 to construct the image display screen 800.

[0089] Here, Figure 8 is a schematic diagram showing an example of the configuration of the image display screen 800 displayed on the administrator terminal display unit 82. The image display screen 800 shown in Figure 8 displays images 206 taken from four mobile devices 6. One of the four mobile devices 6 is highlighted as an enlarged image 804 because the voice communication control operation unit G has been operated and it has entered call mode. In addition, a red frame is displayed around the enlarged image 804 to highlight it. The three mobile devices 6 that are not in call mode are displayed as normal-sized images 802.

[0090] At the right edge of this image display screen 800 is a menu display unit 810, which displays a screen selection button 812, a call selection button 814, and a group call button 816. When the administrator selects the screen selection button 812 from this image display screen 800 displayed on the administrator terminal display unit 82, the selected captured image 206 is enlarged and a call can be made between the mobile device 6 and the administrator terminal 8.

[0091] Furthermore, the administrator can store images 206 (videos, still images) and surrounding audio 208 from the selected mobile device 6 in the business support server 2. When the call selection button 814 is selected, it becomes possible to make a call with the selected mobile device 6. When the administrator can select the group call button 816 displayed on the administrator terminal display unit 82, it becomes possible to make a call with all the displayed mobile devices 6.

[0092] Furthermore, if there are no mobile devices 6 in call mode, for example, all images will be displayed as normal-sized captured images 802, and a maximum of 20 captured images 206 will be displayed per screen of the image display screen 800.

[0093] The display screen providing unit 26 shown in Figure 6 provides the image display screen 800, which is composed of the display screen configuration unit 25, to the administrator terminal 8. When the administrator terminal 8 receives the image display screen 800, it is displayed on the administrator terminal display unit 82 (see Figure 8).

[0094] The voice transmission receiving unit 27 receives the voice transmission from the administrator terminal 8 via the communication line 11 as voice data.

[0095] The call establishment unit 30 establishes a call between the mobile device 6 and the administrator terminal 8 by sending the surrounding voice 208 of the person in charge received by the person in charge surrounding voice reception unit 22 to the administrator terminal 8, and sending the person in charge transmission voice received by the transmission voice reception unit 27 to the mobile device 6. When the mobile device 6 enters call mode, a call becomes possible between the mobile device 6 and the administrator terminal 8.

[0096] Furthermore, the image reception unit L is configured to receive the image transmitted by the person in charge, which is sent as image data via the communication line 11 from the image transmission unit K of the administrator terminal 8.

[0097] Furthermore, the call establishment unit 30 may establish an audio and video call (such as a web conference) between the mobile device 6 and the administrator terminal 8 by sending the surrounding voice 208 of the person in charge received by the person in charge surrounding voice reception unit 22 to the administrator terminal 8, and sending the transmitted voice of the person in charge received by the transmitted voice reception unit 27 to the mobile device 6, and by sending an image display screen including the captured image 206 provided by the display screen provision unit 26 to the administrator terminal 8, and sending the transmitted image of the person in charge received by the transmitted image reception unit L to the mobile device 6.

[0098] The administrator absence determination unit 31 determines whether the administrator operating the administrator terminal 8 is absent when the mobile device 6 switches to call mode. For example, it determines that the administrator is absent if there is no response from the administrator terminal 8 even after a predetermined time has elapsed since switching to call mode, or it determines that the administrator is absent if it detects that an absent button (not shown) on the administrator terminal 8 has been pressed. Multiple mobile devices 61, 6 ２ ..., when the business support system 1 is configured with 6n, multiple portable devices 61, 6 ２ ..., when one of the mobile devices 6i among 6n switches to call mode, the administrator absence determination unit 31 determines that the administrator is absent, and multiple mobile devices 61, 6 ２..., it is controlled so that communication is possible between 6n.

[0099] The behavior information request receiving unit 32 receives behavior information provision requests that are sent as data via the communication line 11 from the behavior information provision request unit 90 of the administrator terminal 8. The behavior information provision unit 33 provides the person in charge behavior information 20 read from the behavior information storage unit 24 to the administrator terminal 8 in response to the behavior information provision request received by the behavior information request receiving unit 32. If the behavior information provision request is made by specifying, for example, a person in charge ID, the person in charge behavior information 200 corresponding to the specified person in charge ID is provided. Also, if, for example, a person in charge ID 202 and acquisition time 210 are specified in combination, the captured image 206 and the person in charge's surrounding audio 208 corresponding to that specification are extracted from the person in charge behavior information 200 and provided. Also, if, for example, a combination of person in charge ID 202, acquisition time 210, and the requested item being a captured image 206 is specified, the captured image 206 corresponding to that specification is extracted from the person in charge behavior information 200 and provided.

[0100] The location information receiving unit 34 receives location information 212 that is sent as data via the communication line 11 from the location information transmission unit 71 of the mobile device 6.

[0101] The map information request unit 35 sends a map information provision request to the map information provision server 10 requesting map information for the area surrounding the location information 212 received by the location information reception unit 34. The map information reception unit 36 ​​receives the map information provided by the map information provision server 10 in response to the map information provision request.

[0102] The behavior information configuration unit 23 configures the person in charge behavior information 200 by linking the location information 212 received by the location information reception unit 34 with the captured image 206 received by the captured image reception unit 21 and the person in charge's surrounding voice 208 received by the person in charge's surrounding voice reception unit 22 (see Figure 7A). Since the location information 212 is linked with the captured image 206 and the person in charge's surrounding voice 208, by specifying the location information 212, the captured image 206 and the person in charge's surrounding voice 208 linked to that location information 212 can be retrieved from the person in charge behavior information 200.

[0103] Furthermore, the display screen configuration unit 25 creates a location information image that displays the location information 212 received by the location information reception unit 34 as the location of the mobile device 6 on the map of the map information received by the map information reception unit 36.The display screen configuration unit 25 then configures the image display screen 800 so that the captured image 206 received by the captured image reception unit 21 and the location information image are displayed on the same screen.The location information image may also display the travel route of the mobile device 6 for the day from the start of work on the map.In addition, the shooting locations of videos and still images taken by operating the video recording button or still image shooting button (or operation buttons B2, B3) displayed on the operation input and video display unit T of the battery unit unit 620Z of the mobile device 6 may be displayed as thumbnails on the map, and when the thumbnail display is selected, the video or still image will be displayed.Thumbnail display refers to the display of sample images of videos and still images.

[0104] The facility floor plan storage unit 37 has facility floor plan information, which includes the layout of the facility, pre-stored therein. When the location information 212 received by the location information reception unit 34 is within the facility, the display screen configuration unit 25 creates a location information image so as to display the location of the mobile device 6 on top of the facility floor plan included in the facility floor plan information. Since the location information 212 also includes height information, the display screen configuration unit 25 determines which floor the mobile device 6 is on within the facility and displays the location of the mobile device 6 on top of the floor plan corresponding to that floor.

[0105] The speech-to-text generation unit 38 generates text from speech. This speech-to-text generation unit 38 generates an ambient speech string 214 from the ambient speech reception unit 22 that receives ambient speech from the person in charge. Existing technologies can be used for this speech-to-text conversion process, and AI (Artificial Intelligence) technology can also be used.

[0106] The behavior information configuration unit 23 configures the person in charge behavior information 200 by linking the surrounding voice string 214 generated by the voice-to-text generation unit 38 to the captured image 206 received by the captured image reception unit 21 and the surrounding voice of the person in charge 208 received by the person in charge surrounding voice reception unit 22 (see Figure 7A). Alternatively, the generated surrounding voice string 214 may be attached to the captured image 206 as a subtitle.

[0107] The abusive language detection unit 40 is configured to detect abusive language. This abusive language detection unit 40 determines whether the surrounding voice string 214 generated by the voice-to-text generation unit 38 contains abusive words that have been registered in advance, and generates abusive language information 216 based on the determination result. The abusive language information 216 includes the abusive words used and their frequency.

[0108] The behavior information configuration unit 23 is configured to include the abusive language information 216 generated by the abusive language determination unit 40 when configuring the person in charge behavior information 200 (see Figure 7A).

[0109] The voice analysis unit 41 analyzes the voice to estimate the mental state. This voice analysis unit 41 analyzes the surrounding voice 208 of the person in charge received by the person in charge surrounding voice reception unit 22 and the surrounding voice string 214 generated by the voice-to-text generation unit 38 to estimate the mental state of the person in charge, and generates person in charge mental state information 218 based on the estimated mental state of the person in charge.

[0110] The voice analysis unit 41 comprehensively analyzes factors such as the volume of the person's voice, the intonation of the voice, the pitch of the voice, the number of sighs per predetermined time, the words used, and the frequency of use of negative words to estimate their mental state. AI technology can also be used in this analysis, and the person's surrounding voice 208 and surrounding voice string 214 are input into a trained generative model to calculate an estimated value of the mental state. The voice analysis unit 41 also compares past and present mental states to determine the stability of the mental state and the direction of change. Furthermore, the voice analysis unit 41 also determines whether the person is in need of mental health care.

[0111] The mental state information 218 for the person in charge is structured to include energy level, voice clarity, accumulated anger level (accumulated anger value), mental state stability, direction (tendency) of changes in mental state, and mental health care need.

[0112] The behavior information structuring unit 23 is configured to include the mental state information 218 of the person in charge generated by the voice analysis unit 41 when structuring the person in charge behavior information 200 (see Figure 7A).

[0113] Here, Figure 7B, as previously described, shows an example of a display screen for the mental state information 218 of the person in charge, which is included in the person in charge's behavior information 200. In Figure 7B, the mental state information 218 of each person in charge is displayed side by side. By displaying such a screen on the administrator terminal 8, the administrator can identify in charge staff who need mental health care at an early stage.

[0114] The facial recognition unit 42 shown in Figure 6 authenticates the face of the person receiving care in the captured image 206 received by the captured image reception unit 21 and identifies the name 304 of the person receiving care. The business support server 2 has feature quantities of the person receiving care's face image registered in advance, and the person receiving care is authenticated by comparing these feature quantities with the feature quantities calculated from the face image captured in the captured image 206. Existing technologies, including AI technology, are used for this facial recognition process. In addition, the person receiving care's name 304 and care recipient ID 302 are registered in the business support server 2 in association with the feature quantities of the person receiving care's face image, so when the facial recognition unit 42 performs authentication of the person receiving care, the person receiving care's name 304 and care recipient ID 302 are identified. When the facial recognition unit 42 identifies the person receiving care's name 304, etc., it generates care recipient identification information 215 including the person receiving care's name 304 and care recipient ID 302.

[0115] The behavior information configuration unit 23 then configures the caregiver behavior information 200 so associates the captured image 206, the caregiver's surrounding voice 208 and surrounding voice string 214 with the care recipient identification information 215 generated by the facial recognition unit 42 (see Figure 7A). With this configuration, for example, it becomes possible to specify the care recipient's name 304 and care recipient ID 302 included in the care recipient identification information 215, and retrieve the captured image 206, etc., in which the specified care recipient is pictured from the caregiver behavior information 200. In other words, when a request for provision of caregiver behavior information 200 (behavior information provision request) is made specifying a combination of caregiver ID 202 and care recipient ID 302, the captured image 206, etc., in which the care recipient corresponding to care recipient ID 302 is pictured is retrieved from the caregiver behavior information 200 corresponding to that caregiver ID 202 and provided.

[0116] Then, the facial expression analysis unit 43 shown in Figure 6 analyzes the facial expression of the person being cared for in the captured image 206 received by the captured image reception unit 21, estimates at least one of emotions such as joy, anger, sadness, or happiness, energy level, and satisfaction level, and calculates an estimated facial expression value 306, which is a numerical representation of the estimation result. Existing technologies, including AI technology, are used in the facial expression analysis process by this facial expression analysis unit 43. For example, feature quantities are calculated from the shape and changes of the eyebrows, the shape and changes of the eyes, cheeks, and lips, the direction of the jaw, etc., and the analysis is performed based on these feature quantities.

[0117] The care recipient information structuring unit 44 configures the care recipient information 300 to include the care recipient's name 304 and care recipient ID 302 identified by the facial recognition unit 42, and the estimated facial expression value 306 obtained by the facial expression analysis unit 43.

[0118] The care recipient information 300 is a collection of information that summarizes the observation records of the care recipient. By specifying the care recipient by name or other means, the observation records of that care recipient can be retrieved from the care recipient information 300.

[0119] Figure 9 shows an example of the configuration of the care recipient information 300. When the care recipient's name 304 and care recipient ID 302 are identified by facial recognition using the captured image 206, the care recipient information 300 is configured to store the identified care recipient's name 304, etc., along with information that constitutes the care recipient's observation record. The components of the care recipient information 300 shown in this figure are just an example, and other elements may be included, or some of these elements may be deleted.

[0120] The care recipient information storage unit 45 shown in Figure 6 stores the care recipient information 300 configured in the care recipient information configuration unit 44.

[0121] The care recipient information request receiving unit 46 receives care recipient information requests sent from the administrator terminal 8 that request care recipient information 300. The care recipient information provision unit 47 provides the care recipient information 300 read from the care recipient information storage unit 45 to the administrator terminal 8 in response to the care recipient information request received by the care recipient information request receiving unit 46. If the care recipient information request is made specifying, for example, a care recipient ID 302, the care recipient information 300 corresponding to the specified care recipient ID 302 is provided. Also, if, for example, a combination of care recipient ID 302 and the requested item being face-to-face time 308 is specified, the face-to-face time 308 corresponding to that care recipient is extracted from the care recipient information 300 and provided.

[0122] The face-to-face time measurement unit 50 measures the face-to-face time 308 during which the person being cared for is visible in the captured image 206 received by the captured image reception unit 21.

[0123] The care recipient information structuring unit 44 is configured to include the care recipient information 300, which is measured by the face-to-face time measurement unit 50 (see Figure 9).

[0124] The image and voice analysis unit 51 performs analysis using AI technology based on the captured image 206 and the surrounding voice 208 of the person in charge. This image and voice analysis unit 51 estimates the person in charge's actions from the captured image 206 received by the captured image reception unit 21 and extracts the person in charge's statements from the surrounding voice 208 received by the person in charge's surrounding voice reception unit 22, and determines the person in charge's work evaluation 220 based on the person in charge's actions and statements. For example, the image and voice analysis unit 51 will give a high evaluation if the person in charge provides care with a smile while speaking kind words to the person in charge, and a low evaluation if the person in charge speaks little, speaks softly, or has a lack of facial expression.

[0125] The behavior information structuring unit 23 is configured to include the employee behavior information 200, which includes the employee's work evaluation 220 determined by the image and sound analysis unit 51 (see Figure 7A). Images 206 of employees who are judged to have excellent work evaluations 220 and those who are judged to have poor work evaluations 220, as well as surrounding voice 208 and surrounding voice strings 214, etc., can be extracted from the employee behavior information 200 and used as training materials for employees or in the creation of manuals.

[0126] Furthermore, the image and sound analysis unit 51 extracts statements that ask questions to the care recipient from the surrounding voice 208 of the care recipient received by the care recipient's surrounding voice reception unit 22, estimates the care recipient's response to the statements from the care recipient's movements captured in the captured image 206 received by the captured image reception unit 21, and determines the care recipient's risk of hearing loss or dementia 310. For example, if the care recipient asks the same question multiple times, or if the care recipient's voice gets louder with each question, the unit will determine that the risk of hearing loss is high. In addition, the risk of dementia is determined by evaluating the time difference between the care recipient's question and the care recipient's response, as well as the consistency of the care recipient's answers to the questions.

[0127] The care recipient information structuring unit 44 is configured to include the care recipient information 300, which includes the risk 310 of hearing loss or dementia of the care recipient as determined by the image and sound analysis unit 51 (see Figure 9).

[0128] The communication unit 52 is configured to send and receive data with the outside world.

[0129] [Operation of the Business Support System] Next, the operation of the business support system 1 according to this embodiment will be described.

[0130] Figure 10 illustrates an example of the operation flow of the business support system 1 performed by the person in charge during their work. As previously stated, in the explanation of Figure 10, the nursing care business is used as an example of the business. In other words, the flow shown in Figure 10 is not limited to the nursing care business. The flow shown in Figure 10 can also be applied to businesses such as construction, childcare and education, security, maintenance, administrative services such as police, ambulance and fire services, trading companies, manufacturing, and medical services. For example, the position of the person receiving care services in the nursing care business can be made to correspond to the person receiving the service in the business to which the business support system according to the present invention is applied, or to other persons in charge of providing said service.

[0131] For example, as shown in Figure 10, the person in charge performs their duties while wearing the portable device 6. The portable device 6 captures images of the surroundings with the camera unit 61, acquires surrounding sounds with the microphone unit 63, and acquires location information with the location information detection unit 70 (step S10). The portable device 6 then sends the acquired captured images 206, the sounds of the person in charge's surroundings 208, and the location information 212, along with the portable device ID that identifies the portable device 6, to the business support server 2 (step S11).

[0132] When the business support server 2 receives the captured image 206, the caregiver's surrounding voice 208, and location information 212 (step S20), the voice-to-text generation unit 38 generates the caregiver's surrounding voice 208 as a surrounding voice string 214 (step S21). The face recognition unit 42 authenticates the person being cared for from the captured image 206, identifies the caregiver's name 304 and caregiver ID 302, and generates caregiver identification information 215 including the caregiver ID 302 (step S22). Subsequently, the action information configuration unit 23 configures the caregiver action information 200 to include the captured image 206, the caregiver's surrounding voice 208, the surrounding voice string 214, location information 212, and caregiver identification information 215 (step S23). In this caregiver action information 200, the captured image 206, the caregiver's surrounding voice 208, and the surrounding voice string 214 are divided into predetermined time intervals, such as 5-minute intervals. Furthermore, the captured image 206, the voice of the person in charge 208, and the voice string 214 are configured to correspond to the acquisition time 210, location information 212, and care recipient identification information 215, respectively. The person in charge action information 200 configured in this way is stored in the action information storage unit 24 (step S24).

[0133] The display screen component 25 processes the captured image 206 to construct an image display screen 800 (step S25), and provides the constructed image display screen 800 to the administrator terminal 8 (step S26). When the administrator terminal 8 receives this image display screen 800 (step S30), the image display screen 800 is displayed on the administrator terminal display unit 82 (step S31).

[0134] The above actions are repeated during the employee's work.

[0135] When the person in charge speaks with the administrator, the voice communication control operation unit G of the headset unit 600 of the mobile device 6 is operated, and the mobile device 6 switches to call mode (step S12). When the business support server 2 detects the notification of switching to call mode (step S27), the call establishment unit 30 operates, enabling a call between the mobile device 6 and the administrator terminal 8 (at this time, as previously described, it may be possible to enable a call that includes video in addition to voice). When the person in charge's voice is detected by the microphone unit 63 of the mobile device 6, the person in charge's voice is sent to the business support server 2 (step S13), and the business support server 2 receives this voice (step S28). At this time, if the area around the person in charge is captured by the camera unit 61 of the mobile device 6, the captured image transmission unit 62 may send an image of the area around the person in charge to the business support server 2, and the captured image receiving unit 21 of the business support server 2 may receive this image of the area around the person in charge.

[0136] The business support server 2 then sends the received audio and the images captured around the person in charge to the administrator terminal 8 (step S29). The administrator terminal 8 outputs the received audio from the audio generation unit 84 (step S32) and reaches the administrator's ears. At this time, the display screen receiving unit 81 of the administrator terminal 8 receives an image display screen from the business support server 2 that displays the images captured around the person in charge. The administrator terminal display unit 82 then displays the image display screen that displays the images captured around the person in charge that were received by the display screen receiving unit 81. This allows the administrator to view the images captured around the person in charge using the administrator terminal display unit 82. Note that the images captured around the person in charge may include not only images captured by the camera unit 61, but also images (photographs), etc., edited by the person in charge on their mobile device 6.

[0137] When the administrator speaks, the voice is input to the voice transmission input unit 85 of the administrator terminal 8 and transmitted from the voice transmission unit 86 to the business support server 2 (step S33). Furthermore, the administrator terminal 8 may also be configured so that, in response to the administrator's operation input, the transmission image transmission unit K transmits the person in charge transmission image input or configured in the transmission image input unit J to the business support server 2. Also, when the business support server 2 receives the administrator's voice (step S200), this voice is transmitted to the mobile device 6 (step S201). At this time, if the business support server 2 receives the person in charge transmission image, this person in charge transmission image may be transmitted from the business support server 2 to the mobile device 6. When the mobile device 6 receives the administrator's voice, it is emitted from the earphone unit 66 (step S14) and reaches the person in charge's ear. The call establishment unit 30 of the business support server 2 controls the voice communication between the mobile device 6 and the administrator terminal 8 to establish a call between the person in charge and the administrator (at this time, as described above, it may be possible to make the call include video in addition to voice).

[0138] When the administrator checks the record of the person in charge's actions, they send a request for action information from the administrator terminal 8 to the business support server 2, specifying the requested items such as the person in charge ID 202 or the person in charge name 204, as well as the acquisition time 210, location information 212, etc. (step S34).

[0139] When the business support server 2 receives this request for action information (step S202), it retrieves the person in charge action information 200 corresponding to this request from the action information storage unit 24 and provides it to the administrator terminal 8 (step S203).

[0140] When the administrator terminal 8 receives the employee action information 200 (step S35), the received employee action information 200 is displayed on the administrator terminal display unit 82 (step S36). Furthermore, when the administrator receives the employee's surrounding voice 208 included in the employee action information 200, it is generated from the voice generation unit 84.

[0141] Here, Figure 11 illustrates an example of a flow in which analysis is performed based on configured personnel activity information 200 in the business support system 1. This example shows a flow in which, after personnel activity information 200 for a predetermined period, such as one day's worth, is accumulated in the business support server 2 through the personnel's work, the personnel activity information 200 obtained during that predetermined period is analyzed all at once. However, this is not the only example, and analysis may be performed in real time using personnel activity information 200 obtained during the personnel's work. Furthermore, as mentioned above, in the explanation of Figure 11, the nursing care business is used as an example of a business. In other words, the business to which the flow shown in Figure 11 can be applied is not limited to the nursing care business. The flow shown in Figure 11 can also be applied to businesses such as construction, childcare and education, security, maintenance, administrative services such as police, ambulance and fire services, trading companies, manufacturing, and medical services. For example, the position of a person receiving care services in a nursing care business can be made to correspond to a person receiving services or other personnel providing such services in the business to which the business support system according to the present invention is applied.

[0142] For example, as shown in Figure 11, first, the business support server 2 reads the already configured employee behavior information 200 from the behavior information storage unit 24 (step S40). Based on the surrounding voice string 214 contained in the read employee behavior information 200, the business support server 2 uses the abusive language determination unit 40 to determine if abusive language is used and generates abusive language information 216 (step S41). Next, the business support server 2 uses the voice analysis unit 41 to generate employee mental state information 218 based on the employee's surrounding voice 208 and surrounding voice string 214 contained in the employee behavior information 200 (step S42). Subsequently, the business support server 2 uses the image and voice analysis unit 51 to determine the employee's work evaluation 220 based on the captured image 206 and employee's surrounding voice 208 contained in the employee behavior information 200 (step S43). The business support server 2 configures the employee behavior information 200 using the behavior information configuration unit 23 so that it includes the obtained abusive language information 216, the employee's mental state information 218, and the employee's work evaluation 220, and stores and updates the configured employee behavior information 200 in the behavior information storage unit 24 (step S44).

[0143] Next, the business support server 2 uses the facial recognition unit 42 to identify the name 304 of the person being cared for, etc., based on the captured image 206 included in the caregiver activity information 200 (step S45). The business support server 2 also uses the facial expression analysis unit 43 to estimate at least one of emotions such as joy, anger, sadness, happiness, energy level, and satisfaction level, and calculates an estimated facial expression value 306, which is a numerical representation of the estimation result (step S46). Subsequently, the business support server 2 uses the face-to-face time measurement unit 50 to measure the face-to-face time 308 based on the captured image 206 included in the caregiver activity information 200 (step S47). After that, the business support server 2 uses the image and sound analysis unit 51 to determine the risk 310 of hearing loss or dementia of the person being cared for, based on the captured image 206 included in the caregiver activity information 200 and the surrounding voice 208 of the caregiver (step S48). Finally, the business support server 2 uses the care recipient information configuration unit 44 to configure care recipient information 300 (step S49) so that it includes the obtained name 304 of the care recipient, estimated facial expression value 306, face-to-face time 308, and the risk of hearing loss or dementia 310 of the care recipient, and stores the configured care recipient information 300 in the care recipient information storage unit 45 (step S50).

[0144] When the administrator checks the record of the person in charge's actions, they send a request for action information from the administrator terminal 8 to the business support server 2, specifying the requested items such as the person in charge's mental state information 218 and work evaluation 220, along with the person in charge's ID 202 or the person in charge's name 204 (step S60).

[0145] When the business support server 2 receives this request for action information (step S51), it retrieves the person in charge action information 200 corresponding to this request from the action information storage unit 24 and provides it to the administrator terminal 8 (step S52).

[0146] When the administrator terminal 8 receives the employee action information 200 (step S61), the received employee action information 200 is displayed on the administrator terminal display unit 82 (step S62).

[0147] When the administrator checks the records of observation of the person receiving care, they specify the requested items such as the time spent together, along with the person receiving care ID 302 or the person receiving care name 304, and send a request for information about the person receiving care from the administrator terminal 8 to the business support server 2 (step S63).

[0148] When the business support server 2 receives this request for care recipient information (step S53), it retrieves the care recipient information 300 corresponding to this request from the care recipient information storage unit 45 and provides it to the administrator terminal 8 (step S54).

[0149] When the administrator terminal 8 receives the care recipient information 300 (step S64), the received care recipient information 300 is displayed on the administrator terminal display unit 82 (step S65).

[0150] Next, the effects of this embodiment will be described.

[0151] According to this embodiment, the actions of the person in charge are recorded by configuring and storing person action information 200, which includes images of the person in charge's surroundings captured by the mobile device 6 and audio of the person in charge's surroundings acquired by the mobile device 6.

[0152] Furthermore, since images captured by the mobile device 6 are displayed on the administrator terminal 8, and communication is possible between the mobile device 6 and the administrator terminal 8, the administrator operating the administrator terminal 8 can provide advice and instructions in real time to the person wearing the mobile device 6, thereby supporting the person's work. In particular, by using the operation input and video display unit T of the mobile device 6, it becomes possible to exchange video instantly and bidirectionally between the person operating the mobile device 6 and the administrator operating the administrator terminal 8 and checking the administrator terminal display unit 82. As a result, the administrator and the person in charge can communicate with each other using video and audio, which allows the administrator to remotely verify the execution of the work and simplifies the creation of work reports by the person in charge.

[0153] As previously described, the operation input and video display unit T enables immediate, bidirectional video and audio communication between the operator (who operates the mobile device 6) and the administrator (who operates the administrator terminal 8 and checks the administrator terminal display unit 82).

[0154] Furthermore, a person wearing the mobile device 6 can remotely participate in online meetings using various applications via the mobile device 6, together with the administrator operating the administrator terminal 8 and other persons wearing the mobile device 6.

[0155] In particular, for example, an operator wearing the mobile device 6 can check a photograph displayed on the operation input / video display unit T and store the photograph in the mobile device storage unit 73. Furthermore, an operator wearing the mobile device 6 can check a video displayed on the operation input / video display unit T and store it in the mobile device storage unit 73.

[0156] For example, the operator can operate the input / video display unit T of the battery unit 620Z of the portable device 6 to create a folder in the portable device storage unit 73 and store information such as videos, photos, and memos in the portable device storage unit 73 of the battery unit 620Z of the portable device 6. The input / video display unit T of the battery unit 620Z of the portable device 6 may also be configured to display a clock so that the operator can check the time.

[0157] Furthermore, in terms of training for the work, information such as videos, photographs, and / or notes related to training and technical transfer for performing the work is stored in the portable device storage unit 73 of the battery unit 620Z of the portable device 6. This allows the person in charge to carry the portable device 6 and check this information at any time by operating the operation input / video display unit T.

[0158] Furthermore, for example, information such as videos, photographs, and / or notes related to education and technical transfer for performing the said work may be stored in the portable device storage unit 73 of the battery unit 620Z of the portable device 6.

[0159] Furthermore, the employee activity information 200 stored on the business support server 2 can be displayed on the administrator terminal 8 to check the past actions of the employee.

[0160] Furthermore, according to this embodiment, since the earphone section 66 is made of bone conduction earphones, the ears of the person wearing the portable device 6 are not blocked by the earphones. The person receives the audio transmitted from the administrator terminal 8 through the bone conduction earphones and can hear the surrounding sounds with their own ears. In this way, both the audio transmitted from the administrator terminal 8 and the surrounding sounds can be heard clearly.

[0161] As previously mentioned, the earphone unit 66 is not limited to bone conduction earphones, but may be other types of earphones such as canal-type earphones or inner-ear earphones.

[0162] Furthermore, according to this embodiment, the camera unit 61, microphone unit 63, and earphone unit 66 are integrated as a headset unit 600, which is worn on the head of the person in charge. As a result, the person wearing the headset unit 600 can use both hands freely. In addition, by adjusting the shooting direction of the camera unit 61 to match the person in charge's line of sight, it is possible to capture images that are close to the person in charge's field of view.

[0163] Furthermore, according to this embodiment, in camera stop mode, the camera unit 61 stops taking pictures and the microphone unit 63 stops acquiring audio. In situations where privacy must be respected, such as during bathing assistance, the system can switch to camera stop mode to stop acquiring images and audio. In addition, a notification sound indicating that the system is in camera stop mode is emitted from the earphone unit 66, so the person in charge can recognize that the system is in camera stop mode.

[0164] Furthermore, according to this embodiment, when the mobile device 6 switches to call mode, it becomes possible to communicate between the mobile device 6 and the administrator terminal 8. Therefore, when the person in charge wants to receive advice or instructions, they can switch to call mode and receive advice from the administrator operating the administrator terminal 8.

[0165] Furthermore, since the captured image 206 from the mobile device 6 being used for the call is highlighted, the administrator operating the administrator terminal 8 can easily recognize the captured image 206 and provide advice or other suggestions.

[0166] Here, as previously described, the call establishment unit 30 of the business support server 2 may establish an audio and video call (such as a web conference) between the mobile device 6 and the administrator terminal 8 by sending the surrounding voice 208 of the person in charge received by the person in charge surrounding voice reception unit 22 to the administrator terminal 8 and sending the person in charge transmission voice received by the transmission voice reception unit 27 to the mobile device 6, and sending an image display screen including the captured image 206 provided by the display screen provision unit 26 to the administrator terminal 8 and sending the person in charge transmission image received by the transmission image reception unit L to the mobile device 6.

[0167] For example, the control unit 60 of the mobile device 6 controls the captured image transmission unit 62, the person in charge's surrounding voice transmission unit 64, the person in charge's voice reception unit 65, the person in charge's image reception unit H, the microphone unit 63, the earphone unit 66, the camera unit 61, and the operation input / video display unit T so that the mobile device 6 and the administrator terminal 8 transmit voice and video bidirectionally via the business support server 2 and the communication line 11. This allows the person in charge's image transmitted from the administrator terminal 8 to be displayed on the operation input / video display unit T and the captured image taken by the camera unit 61 to be transmitted, while also generating the person in charge's voice transmitted from the administrator terminal 8 through the earphone unit 66 and transmitting the person in charge's surrounding voice acquired by the microphone unit 63. This enables voice and video communication (such as web conferencing) between the person in charge operating the mobile device 6 and the administrator operating the administrator terminal 8, enabling efficient support for carrying out the work.

[0168] Thus, the portable device 6 of the business support system 1 makes it possible to secure and record knowledge such as education, knowledge, experience, and case studies necessary for performing the business, protect employees, secure evidence, and improve governance.

[0169] Furthermore, according to this embodiment, the personnel action information 200 is configured such that the captured image 206 and the personnel's surrounding audio 208 are linked to the location information 212, making it possible to identify the location where the image was taken and the location where the audio was acquired.

[0170] Furthermore, since the location of the mobile device 6 is displayed on the map, its location can be easily identified.

[0171] Furthermore, according to this embodiment, the surrounding voice 208 of the person in charge is generated as a surrounding voice string 214, and the person in charge action information 200 is configured to be linked to the captured image 206 and the surrounding voice 208 of the person in charge, so that the voice corresponding to the captured image 206 and the surrounding voice 208 of the person in charge can be recognized as a string. In addition, when creating manuals or record booklets based on the person in charge action information 200, there is no need to transcribe the text, which saves labor.

[0172] Furthermore, according to this embodiment, abusive language information 216 is created based on the result of determining whether or not abusive language is used, and the person in charge's behavior information 200 is configured to include the abusive language information 216. By referring to this abusive language information 216, it is possible to improve work, such as preventing abusive language.

[0173] Furthermore, according to this embodiment, the mental state of the person in charge can be estimated by analyzing their voice, so it can be used for the mental health care of the person in charge.

[0174] Furthermore, according to this embodiment, the face of the person being cared for in the captured image 206 is authenticated to identify the person's name 304, and the person's facial expression is analyzed to estimate at least one of emotions such as joy, anger, sadness, or happiness, energy level, and satisfaction level, and an estimated facial expression value is calculated. By referring to the estimated facial expression value, the customer satisfaction level of each person being cared for can be determined.

[0175] Furthermore, according to this embodiment, the business support server 2 detects the face-to-face time 308 for each person receiving care. This allows for the identification of biases in service provision and can be used for subsequent business improvements.

[0176] Furthermore, according to this embodiment, the business support server 2 performs an analysis based on the captured image 206 and the surrounding voice 208 of the person in charge to determine the person's work evaluation 220. The comparison of images and voices of people with good and bad evaluations can be used for creating training manuals, etc.

[0177] Furthermore, according to this embodiment, by performing an analysis based on the captured image 206 and the surrounding voice 208 of the caregiver, the risk 310 of hearing loss or dementia in the person receiving care can be determined, which can be used for early detection and subsequent care.

[0178] Furthermore, according to this embodiment, when the mobile device 6 switches to call mode, if it is determined that the administrator operating the administrator terminal 8 is absent, multiple personnel wearing the mobile device 6 can communicate with each other, allowing them to receive advice from other personnel. In this case, the business support server 2 may also enable sharing of images captured by the camera unit 61 of the mobile devices 6 of multiple personnel by displaying them on the operation input / video display unit T of each mobile device 6.

[0179] Furthermore, according to this embodiment, the actions of the person in charge are recorded by configuring and storing person action information 200, which includes images of the person in charge's surroundings captured by the mobile device 6 and audio of the person in charge's surroundings acquired by the mobile device 6.

[0180] It should be noted that the present invention is not limited to the above-described embodiments and can be modified as appropriate. For example, the functions of the business support server 2 may be integrated into the administrator terminal 8. Alternatively, the behavior information storage unit 24 and the care recipient information storage unit 45 included in the business support server 2 may be configured as external servers.

[0181] Furthermore, although the business support system 1 according to the above embodiment was described using the nursing care business as an example, it is not limited to the nursing care business and can be applied to other businesses as well. In other words, as previously stated, the business support system according to the present invention can be applied to other businesses such as construction businesses, childcare and education businesses, security businesses, maintenance businesses, administrative businesses such as police, ambulance and fire services, trading companies, manufacturing businesses, and medical businesses.

[0182] For example, when applying this business support system 1 to childcare and education businesses such as nurseries and kindergartens, or to construction businesses, in the case of nursing care businesses, the images 206 captured by the mobile device 6 will show the person being cared for, in the case of childcare and education businesses, the images 206 will show nursery school children and kindergarten children, and in the case of construction businesses, the images will show workers, etc. Then, facial recognition using the captured images 206 will authenticate the nursery school children, kindergarten children, and workers. When the facial recognition unit 42 of the business support server 2 identifies the name and ID of the nursery school children, kindergarten children, and workers, it generates nursery school child identification information and worker identification information corresponding to the care recipient identification information 215 that includes the name and ID.

[0183] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents.

[0184] 1. Business support system 2. Business support server 6 １ , 6 ２ , ..., 6 i , ..., 6 n6. Mobile device 8. Administrator terminal 10. Map information server 11. Communication line 20. Control unit 21. Captured image reception unit 22. Person in charge surrounding voice reception unit 23. Action information configuration unit 24. Action information storage unit 25. Display screen configuration unit 26. Display screen provision unit 27. Transmission voice reception unit 30. Call establishment unit 31. Administrator absence determination unit 32. Action information request reception unit 33. Action information provision unit 34. Location information reception unit 35. Map information request unit 36. Map information reception unit 37. Facility layout storage unit 38. Voice-to-text generation unit 40. Abusive language determination unit 41. Voice analysis unit 42. Face recognition unit 43. Expression analysis unit 44. Care recipient information configuration unit 45. Care recipient information storage unit 46. Care recipient information provision request reception unit 47. Care recipient information provision unit 50. Face-to-face time measurement unit 51 Image and audio analysis unit 52 Communication unit L Transmitted image reception unit 200 Person in charge behavior information 202 Person in charge ID 204 Person in charge name 206 Captured image 208 Person in charge surrounding audio 210 Acquisition time 212 Location information 214 Surrounding audio string 215 Care recipient identification information 216 Abusive language information 218 Person in charge mental state information 220 Performance evaluation 300 Care recipient information 302 Care recipient ID 304 Care recipient name 306 Estimated facial expression value 308 Face-to-face time 310 Risk of hearing loss or dementia 60 Control unit 61 Camera unit 62 Captured image transmission unit 63 Microphone unit 64 Person in charge surrounding audio transmission unit 65 Person in charge transmitted audio reception unit 66 Earphone unit 70 Location information detection unit 71 Location information transmission unit T Operation input and video display unit H Personnel transmission image reception unit 73 Portable device storage unit 74 Communication unit E Battery unit B1 Power button B2, B3 Operation buttons H Personnel transmission image reception unit 600 Headset unit G Voice communication control operation unit 612 Connection unit 614 Widening unit 616 Ear hook unit N1 Communication power cable connection terminal 619 Front unit 620Z Battery unit unit N2 Communication power cable connection terminal 650 Communication power cable 80 Control unit 81 Display screen reception unit 82 Administrator terminal display unit 83 Output voice reception unit 84 Voice generation unit 85 Transmission voice input unit86 Voice transmission unit 90 Action information request unit 91 Action information reception unit 93 Administrator terminal storage unit 94 Care recipient information request unit 95 Care recipient information reception unit 98 Input unit 99 Communication unit 800 Image display screen 802 Normal-size captured image 804 Enlarged captured image 810 Menu display unit 812 Screen selection button 814 Call selection button 816 Group call button

Claims

1. A business support system that monitors and manages the actions of personnel engaged in business and supports the performance of business by said personnel, comprising: a portable device worn by the said personnel; an administrator terminal that displays images taken by the portable device and can communicate with the portable device; and a business support server that constructs and stores personnel action information based on data acquired by the portable device and controls communication between the portable device and the administrator terminal, wherein the portable device and the administrator terminal are connected to the business support server via a communication line, the portable device includes a headset unit, a battery unit, and a communication power cable connecting the headset unit and the battery unit, the headset unit includes: a camera unit that takes pictures of the area around the said personnel and acquires captured images, a microphone unit that receives sounds from the area around the said personnel and generates voice data, which is the sound of the area around the said personnel, an earphone unit that generates sound to be transmitted to the said personnel based on the voice data, and a voice communication control operation unit for the said personnel to operate in order to control the operation of the microphone unit and the earphone unit and to communicate voice data, the battery unit is The portable device comprises: an image transmission unit that transmits images captured by the camera unit to the business support server via the communication line; an ambient sound transmission unit that transmits ambient sound of the person in charge acquired by the microphone unit to the business support server via the communication line; an operation input and video display unit that receives operation input from the person in charge and displays video; an ambient sound reception unit that receives ambient sound data transmitted from the business support server, which is ambient sound for the person in charge to be generated from the earphone unit; an ambient image reception unit that receives ambient image data transmitted from the business support server, which is ambient image for the person in charge to be displayed on the operation input and video display unit; a battery unit that supplies power to the headset unit to operate via the communication power cable, and also supplies power to the battery unit to operate; and a control unit that controls the operation of the portable device.A business support system characterized in that the control unit controls the captured image transmission unit, the person in charge ambient sound transmission unit, the person in charge voice reception unit, the person in charge image reception unit, the microphone unit, the earphone unit, the camera unit, and the operation input / video display unit so that the mobile device and the administrator terminal transmit audio and video bidirectionally via the business support server and the communication line, thereby causing the person in charge image transmitted from the administrator terminal to be displayed on the operation input / video display unit and the captured image captured by the camera unit to be transmitted, and causing the person in charge voice transmitted from the administrator terminal to be generated from the earphone unit and the person in charge ambient sound acquired by the microphone unit to be transmitted.

2. The business support system according to claim 1, characterized in that the image transmitted by the person in charge includes video containing information necessary for the person in charge to perform the said work.

3. The business support system according to claim 2, characterized in that the voice communication by the person in charge includes voice indicating information necessary for the person in charge to perform the said work.

4. The business support system according to claim 1, characterized in that video, photographic, and / or memo information relating to education and / or technical transfer for performing the aforementioned business is stored in the portable device storage unit of the battery unit of the portable device.

5. The business support system according to claim 1, characterized in that the administrator terminal comprises: a display screen receiving unit that receives an image display screen showing the captured image sent from the business support server; an administrator terminal display unit that displays the received image display screen; an output voice receiving unit that receives the surrounding voice of the person in charge sent from the business support server; a voice generating unit that generates the received surrounding voice of the person in charge; a transmission voice input unit that receives the voice transmitted by the person in charge as input; a transmission voice transmission unit that sends the input voice transmitted by the person in charge to the business support server; an action information provision request unit that sends an action information provision request requesting the person in charge's action information to the business support server; an action information receiving unit that receives the person in charge's action information provided from the business support server in response to the action information provision request; a transmission image input unit that receives or constitutes the person in charge's transmission image as input; and a transmission image transmission unit that sends the input or constituted person in charge's transmission image to the business support server.

6. The business support server includes: a captured image receiving unit that receives the captured image sent from the captured image sending unit; a person-in-charge surrounding voice receiving unit that receives the person-in-charge surrounding voice sent from the person-in-charge surrounding voice sending unit; an action information configuring unit that constitutes the person-in-charge action information including the received captured image and the person-in-charge surrounding voice; an action information storage unit that stores the configured person-in-charge action information; a display screen configuring unit that processes the captured image received by the captured image receiving unit to constitute the image display screen; a display screen providing unit that provides the configured image display screen to the administrator terminal; a transmission voice receiving unit that receives the person-in-charge transmission voice sent from the transmission voice sending unit; and a transmission image receiving unit that receives the person-in-charge transmission image sent from the transmission image sending unit K. The business support system according to claim 5, comprising: a call establishment unit that establishes an audio and video call between the mobile device and the administrator terminal by sending the surrounding voice of the person in charge received by the person in charge surrounding voice reception unit to the administrator terminal, sending the person in charge transmission voice received by the transmission voice reception unit to the mobile device, sending an image display screen including the captured image provided by the display screen provision unit to the administrator terminal, and sending the person in charge transmission image received by the transmission image reception unit to the mobile device; an action information request reception unit that receives the action information provision request sent from the action information provision request unit; and an action information provision unit that provides the person in charge action information read from the action information storage unit to the administrator terminal in response to the received action information provision request, wherein when the action information reception unit receives the person in charge action information, the received person in charge action information is displayed on the administrator terminal display unit.

7. The business support system according to claim 1, characterized in that the earphone section is composed of bone conduction earphones, canal-type earphones, or in-ear earphones.

8. The business support system according to claim 1, characterized in that the camera unit, the microphone unit, and the earphone unit are integrated as a headset, and the headset is configured to be worn on the head of the person in charge.

9. The business support system according to claim 6, wherein the portable device has a camera stop mode, in which the camera unit stops taking pictures and the microphone unit stops acquiring audio, and the earphone unit emits a notification sound to indicate that the camera stop mode is in effect.

10. The business support system according to claim 9, characterized in that the mobile device has a call mode, in which a call is possible between the mobile device and the administrator terminal, and the captured image captured by the camera unit is highlighted on the administrator terminal display unit.

11. The mobile device includes a location information detection unit that receives radio waves from a satellite positioning system and calculates location information, and a location information transmission unit that transmits the calculated location information to the business support server, the business support server includes a location information reception unit that receives the location information transmitted from the location information transmission unit, a map information request unit that sends a map information provision request requesting map information for the area surrounding the received location information, and a map information reception unit that receives the map information provided in response to the map information provision request, the activity information configuration unit configures the person in charge activity information so as to link the location information received by the location information reception unit with the captured image received by the captured image reception unit and the person in charge's surrounding voice received by the person in charge's surrounding voice reception unit, the display screen configuration unit creates a location information image that displays the location information received by the location information reception unit as the location of the mobile device on the map of the map information received by the map information reception unit. The business support system according to claim 6, characterized in that the image display screen is configured such that the captured image received by the captured image receiving unit and the location information image are displayed on the same screen.

12. The business support server has a speech-to-text generation unit that generates text from speech, the speech-to-text generation unit generates the surrounding voice of the person in charge received by the person in charge surrounding voice reception unit as a surrounding voice string, and the action information configuration unit configures the person in charge action information so as to link the surrounding voice string generated by the speech-to-text generation unit to the captured image received by the captured image reception unit and the surrounding voice of the person in charge received by the person in charge surrounding voice reception unit, as described in 11.

13. The business support server has an abusive language determination unit that determines abusive language, the abusive language determination unit determines whether the surrounding voice string generated by the voice-to-text generation unit contains abusive language words that have been registered in advance, and generates abusive language information based on the determination result, and the behavior information configuration unit configures the person in charge behavior information to include the abusive language information generated by the abusive language determination unit, the business support system according to 12.

14. The business support server has a voice analysis unit that analyzes voice to estimate mental state, the voice analysis unit analyzes the voice surrounding the person in charge received by the person in charge surrounding voice reception unit and the voice-to-text generation unit to estimate the mental state of the person in charge, and generates person in charge mental state information based on the estimated mental state of the person in charge, and the behavior information configuration unit configures the person in charge behavior information to include the person in charge mental state information generated by the voice analysis unit, the business support system according to 13.

15. The business support server includes: a face recognition unit that authenticates the face of the person to be cared for in the captured image received by the captured image reception unit and identifies the name of the person to be cared for; an expression analysis unit that analyzes the expression of the person to be cared for in the captured image received by the captured image reception unit, estimates at least one of emotions such as joy, anger, sadness, and happiness, energy level, and satisfaction level, and calculates an estimated expression value that quantifies the estimation result; a person to be cared for unit that configures the person to be cared for information including the name of the person to be cared for identified by the face recognition unit and the estimated expression value obtained by the expression analysis unit; a person to be cared for unit that stores the configured person to be cared for unit; a person to be cared for unit that receives a request for provision of person to be cared for from the administrator terminal; and a person to be cared for unit that provides the person to be cared for read from the person to be cared for unit in response to the received request for provision of person to be cared for unit. The administrator terminal is: The business support system according to claim 14, comprising: a care recipient information request unit that sends the care recipient information request to the business support server; and a care recipient information receiving unit that receives the care recipient information provided from the business support server in response to the care recipient information request, wherein when the care recipient information receiving unit receives the care recipient information, the received care recipient information is displayed on the administrator terminal display unit.

16. The business support server has a face-to-face time measurement unit that measures the face-to-face time during which the person to be cared for is visible in the captured image received by the captured image reception unit, and the person to be cared for information configuration unit configures the person to be cared for information to include the face-to-face time measured by the face-to-face time measurement unit, as described in 15.

17. The business support server has an image and sound analysis unit that performs analysis based on the captured image and the surrounding voice of the person in charge, the image and sound analysis unit estimates the actions of the person in charge from the captured image received by the captured image receiving unit, extracts the statements of the person in charge from the surrounding voice of the person in charge received by the surrounding voice receiving unit, and determines the performance evaluation of the person in charge based on the actions and statements of the person in charge, the performance information configuration unit configures the performance information of the person in charge to include the performance evaluation determined by the image and sound analysis unit, the business support system according to 14.

18. The business support server has an image-sound analysis unit that performs analysis based on the captured image and the surrounding voice of the person in charge, the image-sound analysis unit estimates the actions of the person in charge from the captured image received by the captured image reception unit, extracts the statements of the person in charge from the surrounding voice of the person in charge received by the surrounding voice reception unit, and determines the performance evaluation of the person in charge based on the actions and statements of the person in charge, the action information configuration unit configures the person in charge action information to include the performance evaluation determined by the image-sound analysis unit, the image-sound analysis unit extracts statements questioning the person in charge from the surrounding voice of the person in charge received by the surrounding voice reception unit, estimates the person in charge's response to the questioning statements from the actions of the person in charge captured in the captured image received by the captured image reception unit, and determines the risk of hearing loss or dementia of the person in charge, the business support system according to 16.

19. The business support system according to claim 18, wherein the business support server has a plurality of the aforementioned mobile devices, and the business support server has an administrator absence determination unit that determines the absence of the administrator operating the administrator terminal, and when one of the plurality of the aforementioned mobile devices switches to call mode, the administrator absence determination unit determines that the administrator is absent, thereby enabling communication between the plurality of the aforementioned mobile devices.

20. Applicable to a business support system that monitors and manages the actions of personnel engaged in business and supports the performance of business by said personnel, a portable device worn by said personnel, the business support system comprises an administrator terminal that displays images captured by the portable device and can communicate with the portable device, and a business support server that constructs and stores personnel action information based on data acquired by the portable device and controls communication between the portable device and the administrator terminal, the portable device is connected to the business support server via a communication line, the portable device includes a headset unit, a battery unit, and a communication power cable connecting the headset unit and the battery unit, the headset unit includes a camera unit that captures images of the personnel's surroundings, a microphone unit that generates personnel's surrounding voice data when sound from the personnel's surroundings is input, an earphone unit that generates voice to be transmitted to the personnel based on the voice data, and a voice communication control operation unit for the personnel to operate in order to control the operation of the microphone unit and the earphone unit and to communicate voice data, the battery unit is The system includes: an image transmission unit that transmits images captured by the camera unit to the business support server via the communication line; an ambient sound transmission unit that transmits ambient sound of the person in charge acquired by the microphone unit to the business support server via the communication line; an operation input and video display unit that receives operation input from the person in charge and displays video; an ambient sound reception unit that receives ambient sound data transmitted from the business support server, which is ambient sound for the person in charge to be generated from the earphone unit; an ambient image reception unit that receives ambient image data transmitted from the business support server, which is ambient image for the person in charge to be displayed on the operation input and video display unit; and a battery unit that supplies power to the headset unit via the communication power cable for the headset unit to operate and for the battery unit to operate.A portable device comprising a control unit for controlling the operation of the portable device, wherein the portable device and the administrator terminal transmit audio and video bidirectionally via the business support server and the communication line, the control unit controls the captured image transmission unit, the person in charge ambient audio transmission unit, the person in charge communication audio reception unit, the person in charge communication image reception unit, the microphone unit, the earphone unit, the camera unit, and the operation input / video display unit, thereby causing the person in charge communication image transmitted from the administrator terminal to be displayed on the operation input / video display unit and the captured image captured by the camera unit to be transmitted, and causing the person in charge communication audio transmitted from the administrator terminal to be generated from the earphone unit and the person in charge ambient audio acquired by the microphone unit to be transmitted.

Images