Multimodal data collection system

Abstract

This multimodal data collection system comprises: a data provision unit for providing data; a data processing unit for processing the data provided from the data provision unit; and a data storage unit for storing the data processed through the data processing unit, wherein: the data processing unit includes a batch module for dividing a plurality of pieces of data provided from the data provision unit into batch units, and a distributed data processing module for distributing the pieces of batch data divided by the batch module and processing same in parallel; and the distributed data processing module can include a first data processing module for converting a format of the pieces of batch data into a target data format, a second data processing module for converting a protocol of the pieces of batch data into a target data protocol, and a third data processing module for synchronizing the pieces of batch data so that the pieces of batch data can be processed in parallel in different servers.

Description

Multimodal data collection system

[0001] The present invention relates to a multimodal data collection system that collects data of various formats from various sources providing data.

[0002] Recently, with the advancement of wearable sensors, there is a growing need for technology that efficiently utilizes various sensor data.

[0003] It must be possible to secure multimodal data generated through various wearable sensors without errors and maintain data integrity.

[0004] Various clinical participants, or subjects, wear wearable sensors and generate numerous types and diverse data while going about their daily lives, and this data can be utilized in research.

[0005] However, it is not easy to secure a large number of subjects during the research phase, and even if subjects are secured, there may be errors in the subjects' data itself, as well as problems in the data collection and aggregation processes.

[0006] For example, if sensor data regarding a subject's pulse, step count, etc., is to be collected during exercise, sensor data may be collected while the subject is traveling or commuting. There may also be cases where wearable sensors are attached to a pet dog to detect the dog's movements for the purpose of rewarding the subject.

[0007] In order to improve the quality of research using wearable sensor data, it is necessary to resolve issues such as the integrity of sensor data from numerous subjects and the synchronization of multimodal data.

[0008] The technical problem that the present invention aims to solve is to provide a system capable of easily collecting multimodal data.

[0009] Another technical problem that the present invention aims to solve is to provide a system capable of easily processing multimodal data.

[0010] Another technical problem that the present invention aims to solve is to provide a multimodal data acquisition system that is compatible with various devices.

[0011] Another technical problem that the present invention aims to solve is to provide a multimodal data collection system with improved processing efficiency for large-scale data.

[0012] Another technical problem that the present invention aims to solve is to provide a multimodal data collection system with improved data storage efficiency.

[0013] Another technical problem that the present invention aims to solve is to provide a multimodal data collection system with enhanced data security and integrity.

[0014] The technical problems that the present invention aims to solve are not limited to those described above.

[0015] To solve the above technical problem, the present invention provides a multimodal data collection system.

[0016] According to one embodiment, the multimodal data collection system comprises a data providing unit that provides data, a data processing unit that processes data provided from the data providing unit, and a data storage unit that stores data processed through the data processing unit. The data processing unit comprises a batch module that separates a plurality of data provided from the data providing unit into batch units, and a distributed data processing module that distributes and processes the batch data separated from the batch module in parallel. The distributed data processing module may include a first data processing module that converts the format of the plurality of batch data into a target data format, a second data processing module that converts the protocol of the plurality of batch data into a target data protocol, and a third data processing module that synchronizes the plurality of batch data so that the plurality of batch data can be processed in parallel on different servers.

[0017] According to one embodiment, the batch module may determine a batch size based on the amount of data provided from the data provider, and generate the batch data by dividing the plurality of data into batch units based on the determined batch size.

[0018] According to one embodiment, the data providing unit includes a plurality of different data sources, and the plurality of data sources provide data of different formats, and the first data processing module may include converting the format of the plurality of batch data having different formats into the target data format.

[0019] According to one embodiment, the data providing unit includes a plurality of different data sources, and the plurality of data sources provide data of different protocols, and the second data processing module may include converting the protocols of the plurality of batch data having different protocols to the target data protocol.

[0020] According to one embodiment, the data processing unit further includes a data compression module that compresses the batch data processed through the distributed data processing module to generate compressed data, and the data storage unit may include storing the compressed data.

[0021] According to one embodiment, the data compression module may include compressing the batch data using different compression algorithms depending on the type of the batch data.

[0022] According to one embodiment, the data processing unit further includes a data security module that encrypts the batch data processed through the distributed data processing module to generate encrypted data, and the data storage unit may include storing the encrypted data.

[0023] According to one embodiment, the data security module may include generating the encrypted data based on AES (Advanced Encryption Standard) 256-bit encryption.

[0024]

[0025] To solve the above technical problem, the present invention provides a multimodal data collection method.

[0026] According to one embodiment, the multimodal data collection method comprises the steps of: transmitting data from a data provider to a data processing unit; generating a plurality of batch data by dividing a plurality of data transmitted to the data processing unit into batch units; distributing and processing the plurality of batch data in parallel; and storing the processed batch data. The step of distributing and processing the plurality of batch data in parallel may include a first data processing step of converting the format of the plurality of batch data into a target data format; a second data processing step of converting the protocol of the plurality of batch data into a target data protocol; and a third data processing step of synchronizing the plurality of batch data so that the plurality of batch data can be processed in parallel on different servers.

[0027] According to one embodiment, the step of distributing and processing the plurality of batch data in parallel further includes the step of compressing the batch data in which the first to third data processing steps have been performed after the third data processing step, and the step of storing the processed batch data may include storing the compressed data in which the batch data is compressed.

[0028] According to one embodiment, the step of distributing and processing the plurality of batch data in parallel further includes the step of encrypting the batch data for which the first to third data processing steps have been performed after the third data processing step, and the step of storing the processed batch data may include storing encrypted data in which the batch data is encrypted.

[0029]

[0030] To solve the above technical problem, the present invention provides a multimodal data collection / monitoring system.

[0031] According to one embodiment, the multimodal data collection / monitoring system may include a data collection unit that collects sensor data measured from a participant through a sensor and ecological moment evaluation data generated through a survey of the participant; a data processing unit that generates batch data by classifying the sensor data and the ecological moment evaluation data into batch units and processes the generated batch data in parallel by distributing them; a core unit that provides an API for managing the sensor data and the ecological moment evaluation data processed through the data processing unit and stores and manages information about research that the participants can participate in; and a data quality monitoring unit that analyzes and monitors the sensor data and the moment evaluation data processed through the data processing unit.

[0032] According to one embodiment, the data processing unit includes a batch module that separates a plurality of sensor data and ecological moment evaluation data provided from the data collection unit into batch units, and a distributed data processing module that distributes and processes the batch data separated from the batch module in parallel, wherein the distributed data processing module may include a first data processing module that converts the format of the plurality of batch data into a target data format, a second data processing module that converts the protocol of the plurality of batch data into a target data protocol, and a third data processing module that synchronizes the plurality of batch data so that the plurality of batch data can be processed in parallel on different servers.

[0033] According to one embodiment, the core unit includes a remote procedure call management module that provides an API for managing the sensor data and ecological moment evaluation data collected by the data collection unit, a research management module that stores and manages information about research in which the participants can participate, and an analysis management module that corrects the sensor data and ecological moment evaluation data so that visualization of the sensor data and ecological moment evaluation data processed through the data processing unit is achieved, and the data quality monitoring unit may include a visualization module that receives the corrected sensor data and ecological moment evaluation data from the analysis management module and visualizes and displays them.

[0034] A multimodal data collection system according to an embodiment of the present invention can efficiently collect data from various data sources (sensors, smart devices, databases, IoT devices, etc.), process the collected data in real time (data format conversion, data protocol conversion, data synchronization), and safely manage the processed large-scale data.

[0035] Accordingly, compatibility and connectivity among various data sources can be improved, large-scale processing efficiency and performance can be enhanced, storage efficiency and performance can be improved, and data security and reliability can be enhanced.

[0036] As a result, the multimodal data collection system according to the above embodiment can be easily applied to future advanced technology fields such as big data, IoT, and artificial intelligence, and can provide a foundation to accelerate data-based decision-making and service development across industries.

[0037] FIG. 1 is a drawing for explaining each component of a multimodal data collection system according to an embodiment of the present invention.

[0038] FIG. 2 is a drawing for specifically explaining the data processing unit of a multimodal data collection system according to an embodiment of the present invention.

[0039] FIG. 3 is a diagram illustrating an example of data format conversion through a first data processing module of a data processing unit according to an embodiment of the present invention.

[0040] FIG. 4 is a flowchart illustrating a multimodal data collection method according to an embodiment of the present invention.

[0041] FIG. 5 is a flowchart for specifically explaining step S300 of a multimodal data collection method according to an embodiment of the present invention.

[0042] FIG. 6 is a diagram illustrating each configuration of a data collection / monitoring system according to an embodiment of the present invention.

[0043] FIG. 7 is a diagram illustrating exemplary screens for sensor data, EMA data, and sensor data measurement through a smartwatch in a data collection / monitoring system according to an embodiment of the present invention.

[0044] FIG. 8 is a diagram illustrating an exemplary screen of the display status of sensors measured from a smartphone in a data collection / monitoring system according to an embodiment of the present invention.

[0045] FIG. 9 is a diagram illustrating the components of a remote procedure call management module in a data collection / monitoring system according to an embodiment of the present invention.

[0046] The present invention is susceptible to various modifications and may have various embodiments; therefore, specific embodiments are illustrated in the drawings and described in detail in the specific details for carrying out the invention. However, this is not intended to limit the invention to specific embodiments, and it should be understood that the invention includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the invention. Similar reference numerals have been used for similar components in the description of each drawing.

[0047] Terms such as first, second, A, B, etc., may be used to describe various components, but said components should not be limited by said terms. These terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.

[0048] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0049] The terms used in this application are used merely to describe specific embodiments and are not intended to limit the invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "having" are intended to indicate the presence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0050] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this application.

[0051]

[0052] FIG. 1 is a drawing for explaining each component of a multimodal data collection system according to an embodiment of the present invention, FIG. 2 is a drawing for specifically explaining a data processing unit of a multimodal data collection system according to an embodiment of the present invention, and FIG. 3 is a drawing for explaining an example of data format conversion through a first data processing module of a data processing unit according to an embodiment of the present invention.

[0053] Referring to FIGS. 1 and 2, a multimodal data collection system according to the above embodiment may include a data providing unit (100) that provides data, a data processing unit (200) that processes data provided from the data providing unit (100), and a data storage unit (300) that stores data processed through the data processing unit (200). Each component is described below.

[0054] The data providing unit (100) can provide data. According to one embodiment, the data providing unit (100) may include a plurality of data sources, and each data source may provide various data. For example, the data source may include sensors, smart devices, databases, IoT devices, etc. As described above, since the data providing unit (100) provides data from various data sources, the data provided by the data providing unit (100) may have various formats (e.g., JSON, XML, CSV, binary, etc.) and various protocols (e.g., REST (Representational State Transfer), gRPC (Google Remote Procedure Call), HTTP (Hypertext Transfer Protocol), MQTT (Message Queuing Telemetry Transport), CoAP (Constrained Application Protocol), etc.).

[0055] The data processing unit (200) can process a plurality of data provided by the data providing unit (100) in parallel by dividing them into batch units and distributing them. According to one embodiment, the data processing unit (200) may include a batch module (210), a distributed data processing module (220), a data compression module (230), and a data security module (240).

[0056] The batch module (210) can generate multiple batch data by dividing multiple data provided by the data provider (100) into batch units. According to one embodiment, the batch module (210) can determine a batch size based on the amount of data provided by the data provider (100), and generate the batch data by dividing the multiple data into batch units based on the determined batch size. That is, the batch module (210) can adaptively control the batch size according to the amount of data inflow. Accordingly, system resource usage is optimized, and flexible response to data processing of various scales is possible. According to one embodiment, the batch module (210) includes a dynamic batch size adjustment algorithm, a task scheduler, and a resource manager, and can generate the batch data using these.

[0057] The distributed data processing module (220) can distribute and process the batch data separated from the batch module (210) in parallel. According to one embodiment, the distributed data processing module (220) can perform data distribution and parallel processing based on a distributed computing framework such as Apache Hadoop and Apache Spark, and can optimize the distribution process using a distributed query optimizer. Accordingly, data processing speed is improved and linear scalability can be provided.

[0058] According to one embodiment, the data processing module (220) may include first to third data processing modules (221, 222, 223), and the distributed batch data may be processed using the first to third data processing modules (221, 222, 223).

[0059] The first data processing module (221) can convert the format of the batch data into a target data format. That is, the first data processing module (221) can convert batch data having different formats into a single unified data format. According to one embodiment, data formats such as JSON, XML, CSV, binary, etc., can be converted into a JSON data format. For example, the plurality of batch data may have a CSV format as shown in FIG. 3 (a) or a JSON format as shown in FIG. 3 (b). In this case, as shown in FIG. 3 (c), the JSON format can be designated as the target format and then converted from a CSV format to a JSON format. Accordingly, the data processing process can be simplified, allowing for faster data processing. According to one embodiment, the first data processing module (221) may include various data format parsers and converters, as well as API gateways and connector libraries, and can use these to convert the format of the batch data into a target data format.

[0060] The second data processing module (222) can convert the protocol of the batch data into a target data protocol. That is, the second data processing module (222) can convert batch data having different protocols into a single unified data protocol. According to one embodiment, for example, protocols such as REST (Representational State Transfer), gRPC (Google Remote Procedure Call), HTTP (Hypertext Transfer Protocol), MQTT (Message Queuing Telemetry Transport), and CoAP (Constrained Application Protocol) can be converted into the HTTP protocol. Accordingly, seamless communication can be achieved between the data collected and processed through the multimodal data collection system according to the above embodiment and the various data sources described above (sensors, smart devices, databases, IoT devices, etc.). According to one embodiment, the second data processing module (222) may include a protocol conversion engine, a protocol optimization and routing module, and can use these to convert the protocol of the batch data into a target data protocol.

[0061] The third data processing module (223) can synchronize the plurality of batch data so that the plurality of batch data can be processed in parallel on different servers. According to one embodiment, the third data processing module (223) may include a distributed system synchronization algorithm (e.g., Raft or Raxos) and a real-time event processing engine, and can synchronize the plurality of batch data using these. Accordingly, since the consistency of the batch data can be maintained, parallel data processing on various servers can be easily performed.

[0062] The data compression module (230) can generate compressed data by compressing the batch data processed through the distributed data processing module (220). According to one embodiment, the data compression module (230) can compress the batch data using different compression algorithms depending on the type of the batch data. According to one embodiment, the data compression module (230) may include a library of various compression algorithms (e.g., LZ4, Snappy, ZSTD, etc.) and a data type detection and optimal compression selection module, and can compress the batch data using these.

[0063] The data security module (240) can generate encrypted data by encrypting the batch data processed through the distributed data processing module (220). According to one embodiment, the data security module (240) can generate the encrypted data based on AES (Advanced Encryption Standard) 256-bit encryption. Additionally, according to one embodiment, the data security module (240) can verify the integrity of the batch data processed through the distributed data processing module (220) using a blockchain-based data integrity verification system.

[0064] The data storage unit (300) can store data processed through the data processing unit (200). According to one embodiment, the data storage unit (300) includes a distributed storage cluster, and can safely store and manage data using the cluster.

[0065] Consequently, the multimodal data collection system according to an embodiment of the present invention can efficiently collect data from various data sources, process the collected data in real time (data format conversion, data protocol conversion, data synchronization), and safely manage the processed large-scale data. Accordingly, compatibility and connectivity with various data sources can be improved, large-scale processing efficiency and performance can be enhanced, storage efficiency and performance can be improved, and data security and reliability can be enhanced. As a result, the multimodal data collection system according to the above embodiment can be easily applied to future advanced technology fields such as big data, IoT, and artificial intelligence, and can provide a foundation for accelerating data-based decision-making and service development across industries.

[0066] The above describes a multimodal data collection system according to an embodiment of the present invention. The following describes a multimodal data collection method according to an embodiment of the present invention. According to one embodiment, the multimodal data collection method according to the embodiment can be performed through the multimodal data collection system according to the embodiment.

[0067] FIG. 4 is a flowchart for explaining a multimodal data collection method according to an embodiment of the present invention, and FIG. 5 is a flowchart for specifically explaining step S300 of a multimodal data collection method according to an embodiment of the present invention.

[0068] Referring to FIGS. 4 and 5, the multimodal data collection method according to the above embodiment may include the step of transmitting data from a data providing unit to a data processing unit (S100), the step of generating a plurality of batch data by dividing a plurality of data transmitted to the data processing unit into batch units (S200), the step of distributing and processing the plurality of batch data in parallel (S300), and the step of storing the processed batch data (S400). That is, the multimodal data collection process using the multimodal data collection system according to the above embodiment may be carried out through steps S100 to S400.

[0069] According to one embodiment, the step (S300) of distributing and processing the plurality of batch data in parallel may include a first data processing step (S310) of converting the format of the plurality of batch data into a target data format, a second data processing step (S320) of converting the protocol of the plurality of batch data into a target data protocol, and a third data processing step (S330) of synchronizing the plurality of batch data so that the plurality of batch data can be processed in parallel on different servers.

[0070] According to one embodiment, the step (S300) of distributing and processing the plurality of batch data in parallel further includes the step of compressing the batch data in which the first to third data processing steps are performed after the third data processing step (S330), and the step of storing the processed batch data may store the compressed data in which the batch data is compressed.

[0071] According to one embodiment, the step (S300) of distributing and processing the plurality of batch data in parallel further includes the step of encrypting the batch data for which the first to third data processing steps have been performed after the third data processing step (S330), and the step of storing the processed batch data may include storing encrypted data in which the batch data is encrypted.

[0072]

[0073] The above describes a multimodal data collection system and method according to an embodiment of the present invention. Hereinafter, a data collection / monitoring system according to an embodiment of the present invention is described. In describing the data collection / monitoring system according to the above embodiment, 'participant' refers to a person who provides data to the data collection / monitoring system, and 'researcher' may refer to a person who performs research using the data provided to the data collection / monitoring system.

[0074] FIG. 6 is a diagram for explaining each configuration of a data collection / monitoring system according to an embodiment of the present invention, FIG. 7 is a diagram for explaining exemplary screens for sensor data, EMA data, and sensor data measurement through a smartwatch in a data collection / monitoring system according to an embodiment of the present invention, FIG. 8 is a diagram for explaining exemplary screens for displaying the status of sensors measured from a smartphone in a data collection / monitoring system according to an embodiment of the present invention, and FIG. 9 is a diagram for explaining each configuration of a remote procedure call management module in a data collection / monitoring system according to an embodiment of the present invention.

[0075] Referring to FIG. 6, the data collection / monitoring system according to the second embodiment may include a management unit (10), a data collection unit (20), a data processing unit (not shown), a core unit (30), and a DQ monitoring unit (40). Each component is described below.

[0076] The management unit (10) can manage the initiation and configuration of data collection studies by researchers according to research requirements. According to one embodiment, the management unit (10) may include a web-based user interface that helps researchers initiate and configure data collection studies according to research requirements. Researchers can access the data collection / monitoring system through the web-based user interface to create or configure new studies, and participate in existing studies to perform additional configuration and joint monitoring. All researchers participating in the study can manage participants and the data and details provided by said participants, and can invite other researchers to collaborate together.

[0077] For example, Google Sign In for Web, which uses Google's OAuth 2.0 token-based authentication, may be used to allow researchers to easily access the data collection / monitoring system. During the initial authentication process, researchers register on the platform using a Google account, and the data collection / monitoring system can retrieve the researchers' details from a Google server. Accordingly, researchers can easily access the data collection / monitoring system by using the same account after registration. When researchers access the data collection / monitoring system, they can choose the option to start a new study or modify an existing study.

[0078] According to one embodiment, the management unit (10) may include a researcher management module (11), a participant management module (12), and a data source management module (13).

[0079] The above researcher management module (11) can manage the Principal Investigator (PI) to add or remove co-researchers and to grant full or partial authority to co-researchers.

[0080] The above participant management module (12) can manage the invitation, addition, or removal of participants in the study by both the principal investigator and the co-investigators.

[0081] The above data source management module (13) can manage researchers to define and classify data sources (sensor data, ecological moment assessment data, batch data described later) in detail (e.g., classify by column name or data type).

[0082] That is, the above management unit (10) can manage to easily create new research or customize existing research. Accordingly, the research can be quickly set to a state ready for data collection, allowing the research to proceed smoothly.

[0083] The above data collection unit (20) is for data collection and may include a first data collection unit (20a) for collecting sensor data and ecological moment evaluation data and a second data collection unit (20b) for collecting batch data.

[0084] According to one embodiment, the first data collection unit (20a) may include a data collection sensor (21a), a sensor data storage module (22a), an EMA data storage module (23a), and a data integration module (24a).

[0085] The data collection sensor (21a) can measure various data from participants. According to one embodiment, the data collection sensor (21a) may include a smartwatch, a smartphone, or other wearable device (e.g., Polar, Empatica, etc.).

[0086] For example, if the data collection sensor (21a) is a smartwatch, sensor data such as acceleration, gyroscope data, photoplethysmography (PPG) data, and GPS location of the smartwatch can be collected. The measurement of sensor data through the smartwatch can be performed as shown in (c) of FIG. 7.

[0087] In contrast, as another example, if the data collection sensor (21a) is a smartphone, motion data, location-related data, environmental data, activity data, smartphone usage data, and device information can be collected. More specifically, the motion data can be collected from sensors such as accelerometers, gravity sensors, and gyroscopes. Additionally, pre-processed motion information provided by Android (e.g., major movements, number of steps, step detection, etc.) can be collected. The location-related data can be collected from GPS, geofences, scans of nearby BLE devices, magnetic fields, orientations, proximity sensors, etc. The environmental data can be collected from environmental sensors such as illuminance and ambient air pressure. The activity data can be collected using Google APIs, such as activity recognition and activity transition detection. The smartphone usage data can collect device analysis data such as screen on / off / lock / unlock events, battery level and charging status, network connection status, call status, and application usage start / end times. The above device information can collect device-related information such as device model, operating system (OS) version, screen size, device language, and device time zone.

[0088] In contrast, as another example, if the data collection sensor (21a) is a Polar wearable device, various Polar devices such as Polar H9, H10, OH1 and other heart rate sensors can be integrated, and heart rate and motion sensor data can be collected seamlessly by providing plug-and-play functionality.

[0089] In contrast, as another example, if the data collection sensor (21a) is an Empatica wearable device, it can provide a seamless interface with the Empatica E4 wristband and collect various types of data such as heart rate, skin temperature, electrical skin activity (EDA), and motion sensor data.

[0090] The sensor data storage module (22a) can store sensor data measured from the data collection sensor (21a). Alternatively, the EMA storage module (23a) can collect and store Ecological Momentary Assessment (EMA) data generated through a survey of participants. According to one embodiment, the survey for collecting the Ecological Momentary Assessment data may be provided in various ways, such as multiple choice or Likert scales. According to one embodiment, when responses to the survey provided to the participants are collected, data such as the survey completion time, notification details, and survey display time may also be recorded.

[0091] The data integration module (24a) can integrate collected multiple sensor data and instant evaluation data by converting them into the same data representation format. For example, the collected multiple sensor data and instant evaluation data can be integrated by converting them into a JSON string format. According to one embodiment, the multiple sensor data and instant evaluation data can be integrated by converting detailed information such as sensor type, timestamp, and sensor data value into a JSON string format. According to one embodiment, the collected data can be temporarily stored in a local database through an Object Relational Mapping (ORM) framework, converted into an integrated time series format, and then exported in CSV format.

[0092] The second data collection unit (20b) may include a data segmentation module (21b) and a data mapping module (22b).

[0093] The data segmentation module (21b) can generate batch data by converting data collected through other conventional systems and data input by researchers in various formats (CSV, text files, etc.) into a standardized format, and then segment it (classify large files into small sub-files). According to one embodiment, the standardized format may include a JSON string format. The data mapping module (22b) can map and sort the segmented batch data. According to one embodiment, columns and schema fields of data chunks can be sorted according to column names and field names. Additionally, column validity can be verified according to schema definitions and constraints.

[0094] According to one embodiment, regarding the collection cycle, heterogeneous multidimensional data may have significant differences in the collection cycle for each data type. More specifically, unlike Pedometer (step count) and Activity Recongnition (user exercise status) which are collected based on events, HR (Heart rate) and Location (GPS) may be data collected periodically. Even in the case of periodically collected data, heart rate may be collected at 30 Hz and location at 1 Hz. In this case, there is a possibility of confusion regarding data missing. The researcher may need to detect this in a timely manner, check the subject's data collection status, and resolve the issue to ensure that data collection proceeds.

[0095] According to one embodiment, differences in prediction results may occur between different sensors collecting the same data due to collection errors. A smartwatch and a smartphone can recognize the user's exercise status through an Accelometer (ACC), a Gyroscope (GYR), and Gravity (GRV), respectively. In this case, the Activity Sensors collected by the smartwatch and the smartphone may collect different results. The researcher may need to establish a logic to determine which of the two data is true.

[0096] According to one embodiment, regarding collection permission, smartwatches and smartphones may collect data from various internal sensors upon authorization. Since user permission must be obtained for each sensor, data collection through a specific sensor may become impossible if the scope of data collection changes over a certain period. Researchers may need to address this issue by implementing logic to verify and reconfirm permission, or by detecting real-time data collection status by comparing it with other users.

[0097] The data processing unit (not shown) can generate batch data by classifying the sensor data and ecological moment evaluation data collected through the data collection unit (10) into batch units, and can process the generated batch data in parallel by distributing them. According to one embodiment, the data processing unit (not shown) may be identical to the data processing unit (200) of the multimodal data collection system according to the embodiment described with reference to FIGS. 1 to 5. Accordingly, a detailed description is omitted. In addition, the data used in the core unit (30) and DQ monitoring unit (40) described later may be data processed through the data processing unit (not shown). As a result, the reliability of operations performed through the core unit (30) and DQ monitoring unit (40) described later can be improved.

[0098] The core unit (30) is intended to satisfy various research requirements regardless of type of data used in data organization (e.g., sensor data, ecological moment evaluation data, batch data), and the core unit (30) may include a remote procedure call (RPC) management module (31), a research management module (32), a data management module (33), and an analysis management module (34).

[0099] The remote procedure call management module (31) may receive the sensor data, the ecological moment evaluation data, and the batch data processed through the data processing unit (not shown) and provide an API for managing them. According to one embodiment, as shown in FIG. 9, the remote procedure call management module (31) may include an authentication module (31a), a campaign management module (31b), a research participation module (31c), a data transmission module (31d), a statistics module (31e), and a communication management module (31f).

[0100] The authentication module (31a) can manage the use of the data collection / monitoring system by the participant and the researcher. According to one embodiment, the participant and the researcher can register a Google account via Google OAuth 2.0 and use the data collection / monitoring system. Accordingly, the user (participant and researcher) can authenticate easily and securely using a Google account and easily access the data collection / monitoring system.

[0101] The campaign management module (31b) can simplify research management and participation by supporting interaction between the study and the participants. According to one embodiment, the researcher can modify or reorganize the details of the study through the campaign management module (31b) and perform tasks such as adding participants, adjusting data sources (the sensor data, the ecological moment assessment data, the batch data), and uploading files for generating the batch data. Additionally, according to one embodiment, the participant can request a study they can participate in within the data collection / monitoring system through the campaign management module (31b) and access the requested study. Accordingly, the participant can easily identify the study they can participate in.

[0102] The above research participation module (31c) can support a participant in participating in a desired study. According to one embodiment, when a participant is registered by participating in a specific study, the registered participant can upload the sensor data and the ecological moment evaluation data.

[0103] The data transmission module (31d) receives the sensing data, the ecological moment evaluation data, and the batch data from the first data collection unit (20a) and the second data collection unit (20b), and can organize the received data.

[0104] According to one embodiment, data sources provided by participants (the sensing data, the ecological moment evaluation data, and the batch data) can be organized in chronological order. More specifically, a web dashboard retrieves k data samples, and this data window can extend from a specified timestamp to the k-th offset. For example, assuming a specific sensor transmits data at an expected frequency (e.g., 100 Hz), the following conditions may apply to retrieve 1,000 samples (k) from this sensor. If 1,000 samples are collected over a period longer than the expected time (10 seconds), this may indicate missing data. Conversely, if samples are collected within the expected time, this may indicate that the data has been received correctly. The above-described function can play an important role in monitoring the accuracy and completeness of data collection.

[0105] In addition, according to one embodiment, data samples within a specified time period from a start timestamp to an end timestamp can be extracted. The data samples extracted in this way can be efficiently used to analyze application usage data over a day, and can also be easily used to analyze user behavior patterns according to specific time periods or days of the week. As a result, researchers can easily identify trends and changes in time-based data.

[0106] Additionally, according to one embodiment, unprocessed data segments from a participant's data source starting from a specific timestamp may be extracted. For example, raw data segments may be extracted based on a specified timestamp, and the extracted data may be streamed to a client upon request.

[0107] The above statistics module (31e) can provide statistics related to monitoring the online status of a participant and the participant's data upload behavior. According to one embodiment, the statistics module runs in the background and can periodically provide the participant's active status timestamps, thereby allowing verification of whether the participant is currently online. According to one embodiment, the statistics module can retrieve statistics related to data uploaded by the participant. For example, the statistics module can provide information such as the participant's upload frequency, data volume, and last upload time. Accordingly, researchers can effectively verify the participants' activity status and data upload patterns.

[0108] The communication management module (31f) can provide a function to support message exchange between the researcher and the participant and a function to provide notifications. According to one embodiment, the communication management module (31f) can provide a function for direct message exchange between the researcher and the participant. Additionally, according to one embodiment, the communication management module (31f) can provide a search function for unread messages among the messages exchanged between the researcher and the participant. Additionally, according to one embodiment, the communication management module (31f) can provide notifications to all participants in the researcher application or a web-based research management portal. Additionally, according to one embodiment, the communication management module (31f) can provide a search function for notifications that the participant has not yet read.

[0109] The above research management module (32) can store and manage research configuration information so that a researcher can customize a data collection study. According to one embodiment, various parameters related to the research configuration, such as the study name, start and end times, study description, selected data source, and participants, can be selected or modified.

[0110] The data management module (33) can receive and store the sensor data, ecological moment evaluation data, and batch data processed through the data processing unit (not shown), and provide a search function for them.

[0111] According to one embodiment, the data management module (33) may provide a create table function. For example, the create table function may be a function that creates and executes low-level queries within a specific database for participants and data sources. According to one embodiment, the data management module (33) may provide an insert data function. For example, the insert data function may be a function that inserts a single data sample or multiple samples in batch units into a designated table. That is, data can be stored efficiently to meet research requirements. According to one embodiment, the data management module (33) may provide a select first timestamp function. For example, the select first timestamp function may be a function that searches for the timestamp of an initial data sample stored in a database by considering defined filters such as data sources and participant IDs. Accordingly, the starting point of the initial research data can be identified and a specific time-based analysis can be set up. According to one embodiment, the data management module (33) may provide a select last timestamp function. For example, the recent timestamp selection function may be a function that searches for timestamps of recent data samples stored in a database by considering defined filters such as data source and participant ID. Accordingly, monitoring the latest status of data collection or real-time data analysis can be easily performed. According to one embodiment, the data management module (33) may provide a select range function. For example, the select range function may be a function that searches for a setment of time-series data corresponding to between two specified timestamps in a specific participant's data source.According to one embodiment, the data management module (33) may provide a select next k function. For example, the select next k function may be a function to search for a segment consisting of k time series data samples in a data source associated with a specific participant.

[0112] The analysis management module (34) can correct the sensor data, the ecological moment data, and the batch data so that the sensor data, the ecological moment evaluation data, and the batch data processed through the data processing unit (not shown) can be visualized.

[0113] The above data quality (DQ) monitoring unit (40) may include a first monitoring unit (40a) for data analysis and a second monitoring unit (40b) for data visualization.

[0114] The first monitoring unit (40a) can evaluate data quality and verify the consistency and validity of the data. According to one embodiment, the first monitoring unit (40a) can provide the data quality evaluation result, the data consistency evaluation result, and the data validity verification result to the analysis management module (34), and the analysis management module (34) can correct the sensor data, the ecological moment evaluation data, and the batch data based on the received data quality evaluation result, data consistency evaluation result, and data validity verification result.

[0115] According to one embodiment, the first monitoring unit (40a) may include a participant statistics calculation module (41a), a data source statistics calculation module (42a), a data aggregation module (43a), and an anomaly detection module (44a).

[0116] The participant statistics calculation module (41a) can calculate statistics such as data upload frequency, data volume, and time patterns for each participant and provide researchers with information regarding the data collection status for each participant. More specifically, the process of monitoring the data quality of participants through a web dashboard incurs a significant computational load, which may lead to problems such as reduced database operation (storage and retrieval) speed, reduced response speed to users, and real-time performance delays. To address this, the participant statistics calculation module (41a) can pre-calculate key data quality values. That is, the participant statistics calculation module (41a) can solve the aforementioned problems by pre-calculating key data quality values. According to one embodiment, the key data quality values ​​pre-calculated may include quality values ​​for the period during which the participant participated in data collection (participation period), the total amount of data uploaded by the participant (data volume), and time information of the most recently uploaded data (last upload timestamp). For example, the main data quality values ​​are calculated in advance in the participant statistics calculation module (41a), and the calculated main data quality values ​​are provided to the analysis management module (34), and the analysis management module (34) can use them to correct the data.

[0117] The data source statistics calculation module (42a) can evaluate the performance and data collection status of the data sources (the sensor data, the ecological moment evaluation data, the batch data). Accordingly, the researcher can easily monitor the accuracy, response time, and data omissions for each sensor. Since the data source statistics calculation module (42a) can also place a load on the database similarly to the participant statistics calculation module (41a), interaction with the analysis management module (34) can be performed to address this. More specifically, the data source performance result values ​​and data collection status evaluation values ​​calculated through the data source statistics calculation module (42a) can be provided to the analysis management module (34), and the analysis management module (34) can optimize the provided values.

[0118] The data aggregation module (43a) can summarize time-series data to generate pattern information and trend information by time range. Accordingly, researchers can efficiently analyze large-scale data. According to one embodiment, the data aggregation module (43a) can calculate useful statistics from time-series data. For example, it can calculate various statistical indicators such as mean, median, minimum, maximum, and sum, and thereby summarize the key characteristics of the data. In addition, the data aggregation module (43a) can identify outliers or specific patterns by understanding the overall distribution of the data. For example, if a participant consistently reports a stress level in a survey for ecological moment assessment, that response may be treated as an inattentive response. Furthermore, if the stress level reported in the survey for ecological moment assessment shows a consistent pattern over time, it may be treated as having a problem with data reliability. The data aggregation module (43a) can quickly detect the problematic data.

[0119] The above anomaly detection module (44a) can analyze data to identify missing values, abnormal values, or abnormal data patterns. Accordingly, researchers can correct abnormal data early.

[0120] The second monitoring unit (40b) may include a visualization module (41b) and a problem reporting module (42b).

[0121] The visualization module (41b) can receive the corrected sensor data, ecological moment evaluation data, and batch data from the analysis management module (34) and visualize and display them. Accordingly, researchers can easily identify missing data and abnormal behavior of participants.

[0122] According to one embodiment, data quality can be visualized. More specifically, it can be visualized to provide insights into the completeness, uniqueness, and timeliness of the data. For example, completeness may represent the ratio of the actual amount of data uploaded to the expected amount of data within a specific time range. For example, uniqueness may represent filtering out identical data that is repeatedly uploaded and retaining only unique data. For example, timeliness may represent an evaluation of whether the collected data is accessible within a certain period after it is generated.

[0123] Additionally, according to one embodiment, individual participants may be visualized. For example, individual participants' behavioral patterns, such as movement paths, compliance with ecological moment assessments, and other behavioral indicators, may be visualized to be summarized. Additionally, according to one embodiment, relationships between participants may be visualized. For example, the distribution of specific data source values ​​among various participants may be visualized to be compared.

[0124] The problem reporting module (42b) can systematically verify participant statistics, data source statistics, and data outliers. According to one embodiment, the problem reporting module (42b) can generate an automated email report highlighting problematic participants and the problems during the verification process, and the generated report can be sent to the researcher. This feature can be usefully applied in large-scale studies where it is difficult to manually track problematic participants. The report sent to the researcher may include details such as the amount of anomalous data (e.g., abnormal or missing data) and the last data submission time for specific data sources and participants.

[0125]

[0126] Although the invention has been described with reference to the above embodiments, those skilled in the art will understand that various modifications and changes can be made to the invention without departing from the spirit and scope of the invention as set forth in the following claims.

[0127] The present invention can be used in the big data industry, IoT industry, and artificial intelligence industry.

Claims

1. A multimodal data collection system comprising a data providing unit that provides data, a data processing unit that processes data provided from the data providing unit, and a data storage unit that stores data processed through the data processing unit, The data processing unit comprises a batch module that separates a plurality of data provided from the data providing unit into batch units, and a distributed data processing module that distributes and processes the batch data separated from the batch module in parallel. A multimodal data collection system comprising a distributed data processing module, a first data processing module that converts the format of a plurality of batch data into a target data format, a second data processing module that converts the protocol of the plurality of batch data into a target data protocol, and a third data processing module that synchronizes the plurality of batch data so that the plurality of batch data can be processed in parallel on different servers.

2. In Paragraph 1, A multimodal data collection system comprising a batch module that determines a batch size according to the amount of data provided from the data provider, and generates the batch data by dividing the plurality of data into batch units based on the determined batch size.

3. In Paragraph 1, The above data providing unit includes a plurality of different data sources, and the plurality of data sources provide data of different formats, wherein A multimodal data collection system comprising the first data processing module converting the formats of a plurality of batch data having different formats into the target data format.

4. In Paragraph 1, The above data providing unit includes a plurality of different data sources, and the plurality of data sources provide data of different protocols, wherein A multimodal data collection system comprising a second data processing module that converts the protocols of a plurality of batch data having different protocols into the target data protocol.

5. In Paragraph 1, The above data processing unit further includes a data compression module that compresses the batch data processed through the distributed data processing module to generate compressed data, and A multimodal data collection system comprising a data storage unit that stores the compressed data.

6. In Paragraph 5, A multimodal data collection system comprising a data compression module that compresses the batch data using different compression algorithms depending on the type of the batch data.

7. In Paragraph 1, The above data processing unit further includes a data security module that encrypts the batch data processed through the distributed data processing module to generate encrypted data, and A multimodal data collection system comprising a data storage unit that stores the encrypted data.

8. In Paragraph 7, A multimodal data collection system comprising a data security module that generates the encrypted data based on AES (Advanced Encryption Standard) 256-bit encryption.

9. A step of transmitting data from the data provider to the data processing unit; A step of generating multiple batch data by dividing multiple data transmitted to the data processing unit into batch units; A step of distributing multiple batch data and processing them in parallel; and The step of storing processed batch data is included, The step of distributing and processing the above plurality of batch data in parallel is A first data processing step for converting the format of the plurality of batch data above into a target data format; A second data processing step for converting the protocols of the plurality of batch data above into a target data protocol; and A multimodal data collection method comprising a third data processing step for synchronizing the plurality of batch data so that the plurality of batch data can be processed in parallel on different servers.

10. In Paragraph 9, The step of distributing and processing the plurality of batch data in parallel further includes the step of compressing the batch data for which the first to third data processing steps have been performed after the third data processing step, and A multimodal data collection method comprising the step of storing the processed batch data, which includes storing compressed data in which the batch data is compressed.

11. In Paragraph 9, The step of distributing and processing the plurality of batch data in parallel further includes the step of encrypting the batch data for which the first to third data processing steps have been performed after the third data processing step, and A multimodal data collection method comprising the step of storing the processed batch data, wherein the batch data is encrypted and the encrypted data is stored.

12. A data collection unit that collects sensor data measured from a participant through a sensor, and ecological moment evaluation data generated through a survey of said participant; A data processing unit that separates the sensor data and the ecological moment evaluation data into bath units to generate bath data, and distributes and processes the generated bath data in parallel; A core unit that provides an API for managing the sensor data and ecological moment assessment data processed through the data processing unit, and stores and manages information about research that the participants can participate in; and A data collection / monitoring system comprising a data quality monitoring unit that analyzes and monitors the sensor data and instantaneous evaluation data processed through the data processing unit.

13. In Paragraph 12, The data processing unit comprises a batch module that separates a plurality of sensor data and ecological moment evaluation data provided from the data collection unit into batch units, and a distributed data processing module that distributes and processes the batch data separated from the batch module in parallel. The distributed data processing module comprises a first data processing module that converts the format of a plurality of batch data into a target data format, a second data processing module that converts the protocol of the plurality of batch data into a target data protocol, and a third data processing module that synchronizes the plurality of batch data so that the plurality of batch data can be processed in parallel on different servers, thereby forming a data collection / monitoring system.

14. In Paragraph 12, The above core part is, A remote procedure call management module that provides an API for managing the sensor data and ecological moment evaluation data collected by the data collection unit; A research management module that stores and manages information on research that the above participants are eligible to participate in; and It includes an analysis management module that corrects the sensor data and the ecological moment data so that the sensor data and the ecological moment evaluation data processed through the data processing unit can be visualized. The above data quality monitoring unit is, A data collection / monitoring system comprising a visualization module that receives and visualizes the sensor data and ecological moment evaluation data corrected from the analysis management module.

Images