Data analysis device and data analysis method

Abstract

A data analysis device according to the present disclosure comprises: a communication circuit; a memory for storing at least one instruction; and at least one processor including a processing circuit for executing the at least one instruction, wherein the at least one instruction causes, when executed individually or collectively by the at least one processor, the data analysis device to: establish a first connection connected to a data warehouse; generate a temporary table in the data warehouse through the first connection; store data, which is acquired from at least one table of the data warehouse, in the temporary table through the first connection; establish, through the first connection, a second connection connected to a cloud storage; store a temporary file, which includes information of the temporary table, in the cloud storage through the second connection; establish a third connection connected to the cloud storage; acquire a file including information of the temporary file from the cloud storage through the third connection; convert the file into data; and analyze the data.

Description

Data analysis device and data analysis method

[0001] The present disclosure relates to a data analysis device and a data analysis method.

[0002] Recently, cloud services have been evolving beyond the simple rental of IT resources into platforms that integrate innovative technologies and create new value. In particular, with the advancement of serverless computing, services are expanding beyond simple function execution to include a variety of offerings such as container-based serverless services, database services, and storage services.

[0003] For example, Google Cloud Platform (GCP) is a cloud computing platform provided by Google that can offer various services to enable developers and enterprises to build, deploy, and operate applications. GCP includes services based on IaaS (Infrastructure as a Service), PaaS (Platform as a Service), and SaaS (Software as a Service), and specifically, it can provide various services in a cloud environment such as virtual machine services (Compute Engine), storage services (Cloud Storage), application deployment services (App Engine), serverless development environments (Cloud Functions), document storage and editing (Google Drive, Google Docs), and data warehouses (DWH).

[0004] A data warehouse is a centralized data repository for storing, analyzing, and managing large volumes of data. It is a decision support system that stores data extracted, transformed, and integrated from various operational systems within an enterprise (e.g., order processing systems, customer relationship management systems). Unlike operational systems (OLTP, online transaction processing) that process real-time transaction data, it corresponds to an OLAP (online analytical processing) service designed to assist executives in making strategic decisions by analyzing historical data multidimensionally and identifying trends. Google's BigQuery, Amazon's Redshift, and Snowflake are representative cloud-based data warehouses that support standard SQL (structured query language) and can provide services optimized for big data analysis, artificial intelligence (AI), and machine learning (ML).

[0005] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0006] Data can be acquired from a data warehouse and used as a target for analysis in a data analysis environment. However, directly acquiring data from a data warehouse has the disadvantage of being time-consuming, or additional costs may be incurred when acquiring large volumes of data. For example, there may be free service options that are time-consuming but incur no additional costs, and paid service options that guarantee fast execution times; the free service option may take more than 50 times longer than the paid service option.

[0007] In addition, when a data analysis environment is provided as a cloud service, the size of the resources (CPU, GPU, NPU, TPU, memory, storage) required by the data analysis environment may vary depending on the size of the data to be analyzed or at least one of the analysis algorithms, so an adaptive resource allocation device may be required to build an efficient system.

[0008] According to the present disclosure, data can be acquired efficiently by considering the trade-off between time and cost, and resources of the data analysis environment can be dynamically allocated by a resource allocation device.

[0009] One embodiment of the present disclosure may provide a data analysis device and a data analysis method.

[0010] A data analysis device according to the present disclosure comprises a communication circuit; a memory for storing at least one instruction; and at least one processor including a processing circuit for executing said at least one instruction, wherein the at least one instruction may cause the data analysis device to perform at least one operation when executed individually or collectively by said at least one processor.

[0011] The above at least one operation may include an operation of establishing a first connection connected to a data warehouse. The above at least one operation may include an operation of creating a temporary table in the data warehouse through the first connection.

[0012] The above at least one operation may include the operation of storing data obtained from at least one table of the data warehouse in the temporary table through the first connection.

[0013] The above at least one operation may include an operation of establishing a second connection connected to cloud storage through the first connection.

[0014] The above at least one operation may include the operation of storing a temporary file containing information of the temporary table in the cloud storage through the second connection.

[0015] The above at least one operation may include an operation to establish a third connection connected to the cloud storage.

[0016] The above at least one operation may include an operation of receiving the temporary file from the cloud storage through the third connection and obtaining the first data file.

[0017] The above at least one operation may include an operation of converting the file into data.

[0018] A data analysis method according to the present disclosure may include at least one of the following: connecting to a data warehouse; creating a temporary table in the data warehouse; storing data obtained from at least one table of the data warehouse in the temporary table; obtaining metadata related to the data stored in the temporary table; obtaining resources necessary for data analysis based on the metadata; connecting to cloud storage through the data warehouse; transmitting information of the temporary table to the cloud storage and storing it as a file; connecting to the cloud storage from the data processing device; obtaining a file from the cloud storage; converting the file into data; and analyzing the data.

[0019] A computer-readable medium according to the present disclosure may store a program that commands the execution of a data analysis method comprising at least one of the following operations: connecting to a data warehouse; creating a temporary table in the data warehouse; storing data obtained from at least one table of the data warehouse in the temporary table; obtaining metadata related to the data stored in the temporary table; obtaining resources necessary for data analysis based on the metadata; connecting to a cloud storage through the data warehouse; transmitting information of the temporary table to the cloud storage and storing it as a file; connecting to the cloud storage from the data processing device; receiving a temporary file from the cloud storage and obtaining a first data file; converting the file into data; and analyzing the data.

[0020] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0021] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments of the present disclosure.

[0022] FIG. 2 is a system configuration diagram of a user device and a cloud platform according to one embodiment of the present disclosure.

[0023] FIG. 3 is a system configuration diagram of a user device and a cloud platform according to one embodiment of the present disclosure.

[0024] FIG. 4 is a flowchart for explaining a data analysis method according to one embodiment of the present disclosure.

[0025] FIG. 5 is a flowchart illustrating a data analysis method according to one embodiment of the present disclosure.

[0026] FIG. 6 is a flowchart illustrating a data analysis method according to one embodiment of the present disclosure.

[0027] FIG. 7 is a flowchart illustrating a data analysis method according to one embodiment of the present disclosure.

[0028] FIG. 8 is a flowchart illustrating a data analysis method according to various embodiments of the present disclosure.

[0029] In the following description, the attached drawings are referenced, and specific examples of implementation are illustrated within the drawings. Additionally, other examples may be used and structural modifications may be made without departing from the scope of the various examples.

[0030] Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings so that those skilled in the art can easily practice them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0031] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments of the present disclosure.

[0032] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0033] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0034] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence is performed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0035] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0036] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0037] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0038] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0039] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0040] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0041] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0042] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0043] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0044] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0045] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0046] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0047] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0048] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a communication module (192) (e.g., cellular communication module, short-range communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0049] The communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for eMBB realization, loss coverage (e.g., 164 dB or less) for mMTC realization, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for URLLC realization.

[0050] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197). According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0051] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface) and exchange signals (e.g., commands or data) with each other.

[0052] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0053] The number of processors (120) may be one or more. For example, the processor (120) may have the structure of a multi-core processor such as a dual core, a quad core, or a hexa core.

[0054] The processor (120) can control the operations of the electronic device (101) by executing instructions stored in the memory (130). For example, the processor (120) may correspond to a plurality of processors that divide and collectively perform a plurality of operations among the processors.

[0055] FIG. 2 is a system configuration diagram of a user device and a cloud platform according to one embodiment of the present disclosure.

[0056] Referring to Fig. 2, the user device performs the role of connecting to a cloud platform, and most operations are performed using resources provided by the cloud platform.

[0057] According to one embodiment, a data warehouse provided on a cloud platform (e.g., Google BigQuery) can process petabyte-scale queries in minutes and can be used to store and process large volumes of data. An essential step in data analysis is to configure a data analysis environment with sufficient resources (e.g., CPU, GPU, NPU, TPU, memory) to load and analyze data, and to load data to be analyzed from the data warehouse into the analysis environment. Data warehouses are typically operated on a usage-based billing basis and can provide free services with no additional costs, as well as paid services that offer improved computing power and data processing speeds compared to free operations.

[0058] According to one embodiment, paid operations provide a service to enable rapid data analysis, and incur costs based on the size of the data being processed. Free operations incur no additional costs, but the time required to perform data analysis may be longer compared to paid operations.

[0059] According to one embodiment, in addition to a data warehouse (251), the cloud platform provides a storage service (253) that stores and manages data among various services. For example, Google Cloud Storage (GCS) is a scalable, high-performance, and highly durable object storage service provided by GCP, used to store and access various types of structured and unstructured data, ranging from website content to backup data and big data analysis data. GCS is highly durable and has no limitations on storage capacity or throughput, so it can be used without problems even as the amount of data increases. It also has high availability to allow access to data anytime and anywhere, and can provide various storage classes according to data access frequency and cost requirements. Furthermore, Google Cloud Storage can be integrated with other GCP services such as Google Compute Engine, Google Kubernetes, and Google BigQuery to be utilized in building various solutions. Cloud storage may be referred to as network storage.

[0060] According to one embodiment, a user device (210) can access a cloud platform (250) according to the present disclosure to perform data analysis work. The cloud platform (250) may include at least one of a data warehouse (251), cloud storage (253), a data analysis device (255), a resource calculation device (257), or a resource allocation device (259), and each of the user device (210), the data warehouse (251), the cloud storage (253), the data analysis device (255), the resource calculation device (257), and the resource allocation device (259) may be configured to include at least one electronic device (e.g., the electronic device (101) of FIG. 1) and one or more servers (e.g., the server (108) of FIG. 1). When each of the user device (210), data warehouse (251), cloud storage (253), data analysis device (255), resource calculation device (257) and resource allocation device (259) is configured as any one of (1) a plurality of electronic devices (101), (2) a plurality of servers (108), or (3) a system including all of one or more electronic devices (101) and one or more servers (108), the system including (1) a plurality of electronic devices (101), (2) a plurality of servers (108), or (3) a system including all of one or more electronic devices (101) and one or more servers (108) may be connected through a first network (e.g., the first network (198) of FIG. 1) and / or a second network (e.g., the second network (199) of FIG. 1).

[0061] According to one embodiment, the user device (210) may be any one of a PC (personal computer), a workstation, a virtual desktop infrastructure (VDI), a virtual machine (VM), or a desktop as a service (DaaS).

[0062] According to one embodiment, the data analysis device (255) is an electronic device capable of processing large amounts of data, and may be configured as a cloud system and may be allocated resources (CPU, GPU, NPU, TPU, memory, storage) according to the scale of data analysis.

[0063] According to one embodiment, Python is frequently used as a programming language for recent data analysis, and the data analysis device (255) may provide a data analysis environment such as JupyterLab, Jupyter Notebook, and Colab that supports the Python language. The data analysis device (255) may be provided as a cloud service and may support multi-tenancy so that multiple users may be allocated resources and perform data analysis using all or part of the provided cloud service resources.

[0064] According to one embodiment, the Python language provides various libraries for data analysis, and specifically, various libraries such as pandas, which provides data manipulation and analysis; scikit-learn, which provides machine learning and statistical modeling; statmodels, which is a library specialized in statistics; scipy, which provides basic statistics and probability distribution functions; and numpy, which provides numerical computation, vector and matrix operation functions, can be used.

[0065] According to one embodiment, the data analysis device (255) may be composed of a single machine or may be a virtual device provided as a cloud service by connecting multiple machines. The cloud service may be a service where, when resources such as servers or storage are needed, they are rented from vendors such as Google, Amazon, or Microsoft, and costs are paid for the amount used. The data analysis device provided as a cloud service may receive resources for data analysis from a resource distribution device, and the resources may include at least one of a CPU, GPU, NPU, TPU, memory, and storage. Additionally, the resources may be allocated as a portion of the resources of a single device, or multiple resources may be allocated from multiple devices. If the memory of the data analysis device is insufficient, it may be difficult for the data analysis device to process the data. In the present disclosure, the term "data analysis device" may be used interchangeably with "analysis environment," "data analysis environment," and "Python analysis environment."

[0066] According to one embodiment, a data warehouse (251) refers to a data analysis tool that supports standard SQL (structured query language) and is used to store and analyze large amounts of data. It provides related internal corporate data, such as CRM systems, ERP systems, and marketing systems operated by the company, as a database, and enables historical analysis by storing past data in a time series. Google BigQuery, Amazon RedShift, and Snowflake are representative data warehouse services, and a usage-based billing system is applied. In addition, the data warehouse can provide free and paid operations for data querying.

[0067] According to one embodiment, free operations may be provided free of charge as a REST API-based service, but they may be slow. The operation of extracting processed data externally may take a particularly long time. Paid operations are faster, but costs may be incurred. The amount of data serving as the basis for the cost may refer to the size of the source data that is the subject of the query. Due to the nature of big data, the source table may be tens of terabytes.

[0068] According to one embodiment, based on a data query, an operation to extract only data corresponding to user-specified conditions from a table storing various data may be performed. Additionally, the extracted data may be transmitted to a designated storage, such as cloud storage.

[0069] According to one embodiment, cloud storage (253) refers to a cloud service that allows data to be stored and accessed via the Internet. Storage can be flexibly increased according to usage, and may include functions such as automatic backup, version control, and deletion recovery. Representative cloud storage services exist, such as Google’s Google Drive, Google Cloud Storage, AWS’s Amazon S3, Apple’s iCloud Drive, Microsoft’s OneDrive, and Dropbox Inc.’s Dropbox.

[0070] According to one embodiment, the resource calculation device (257) refers to an electronic device that calculates the amount of resources required by the data analysis device (255). The data analysis device (255) can calculate the resources required for data analysis based on at least one of the size and structure of the data to be analyzed, the analysis method, and the analysis algorithm. The resource calculation device (257) may be configured as a separate device, may be configured together with a resource allocation device in a single device, or the data analysis device may perform the resource calculation function.

[0071] According to one embodiment, the resource allocation device (259) is a device capable of allocating resources based on the result of resource calculation, and the resource allocation device (259) can allocate the necessary resources to the data analysis device (255).

[0072] FIG. 3 is a system configuration diagram of a user device and a cloud platform according to one embodiment of the present disclosure.

[0073] According to FIG. 3, data analysis can be performed using the first cloud platform (340) and the second cloud platform (370). For example, the operators of the first cloud platform (340) and the second cloud platform (370) may be different from each other, the first cloud platform (340) may be a cloud service commercially provided by a service provider, and the second cloud platform (370) may be a cloud service built directly by a user.

[0074] According to the present disclosure, the first cloud platform (340) may include at least one of a data warehouse (341) or a cloud storage (343), and the second cloud platform (370) may include at least one of a resource calculation device (373), a resource allocation device (375), or a data analysis device (371). The data analysis device (371) may utilize a Python analysis environment. For a general description of the data warehouse (341), cloud storage (343), data analysis device (371), resource calculation device (373), and resource allocation device (375), refer to the description of the data warehouse (251), cloud storage (253), data analysis device (255), resource calculation device (257), and resource allocation device (259) of FIG. 2.

[0075] According to the present disclosure, a data analysis device (371) can establish a connection with a data warehouse (341) (e.g., Google BigQuery). Additionally, the data analysis device (371) can be connected to a cloud storage (343) while connected to the data warehouse (341). Additionally, the data analysis device (371) can be directly connected to the cloud storage (343). The data analysis device (371) can acquire data directly from the data warehouse (341) or acquire data from the data warehouse (341) via the cloud storage (343). When the data analysis device (371) acquires data directly from the data warehouse (341), it may take a long time or incur a high cost. On the other hand, it may be possible to quickly transfer files from the data warehouse (341) to the cloud storage (343) at a low cost. It may be more economical in terms of cost and time for the data analysis device (371) to acquire files from cloud storage (343) via cloud storage (343) rather than acquiring data directly from the data warehouse (341). That is, it may be more efficient for the data analysis device (371) to acquire data from cloud storage (343) after transferring data from the data warehouse (341) to cloud storage (343) rather than the data analysis device (371) acquiring data directly from the data warehouse (341).

[0076] According to one embodiment, when data is stored in a temporary table of a data warehouse (341), meta-information related to the temporary table of the data warehouse (341) may be generated. Based on the meta-information related to the temporary table, a data analysis device (371) or a resource calculation device (373) may calculate resources for data analysis. Based on the calculated resources, a resource allocation device (375) may redistribute resources to the data analysis device (371).

[0077] The resource calculation device (373) according to the present disclosure can calculate the amount of resources required by the data analysis device based on meta-information, and the resource distribution device (375) can distribute the resources required by the data analysis device according to the resources calculated by the resource calculation device (373). The resource calculation device (373) and the resource distribution device (375) according to the present disclosure may be configured as separate devices, and a single device may perform both the roles of the resource calculation device (373) and the resource distribution device (375). Additionally, the resource calculation device (373) and the resource distribution device (375) may be configured as cloud services. Furthermore, the data analysis device (371) may also perform the role of the resource calculation device (373).

[0078] According to one embodiment, the data analysis device (371) can create a temporary table in the user space within the data warehouse (341) to extract data from the data warehouse (341), and store the processed data in the created temporary table.

[0079] According to one embodiment, data stored in a temporary table of a data warehouse (341) may be transmitted and stored as a temporary file in cloud storage (343). The temporary file may be stored in cloud storage (343) with the data structure of the processed data intact, or the temporary file may be stored in cloud storage (343) after the data structure of the processed data has been converted. Data stored in the temporary table may be converted in the data warehouse (341), then transmitted to cloud storage (343) and stored as a temporary file. Additionally, data stored in the table may be converted in cloud storage (343) after being transmitted to cloud storage (343) and stored as a temporary file. The stored temporary file may be stored in various formats such as CSV, Excel, SQL, database dump files, and JSON. The stored file may be stored in a split format or without splitting. Each stored file may be stored in a compressed form or in an uncompressed form. Additionally, the stored file may be in a row-based format or a column-based format. Since similar data tends to be repeated in the same column of a table, the compression rate of column-based data can be higher than that of row-based data. As data warehouses often utilize column-based databases, data obtained from data warehouses may be in a column-based format. When data obtained from data warehouses is in a column-based format, the compression rate of temporary files stored in cloud storage can be high.

[0080] According to one embodiment, a data analysis device (371) can obtain a temporary file stored in cloud storage (343). The obtained temporary file can be converted into various data formats for data analysis. For example, the obtained temporary file can be converted into a DataFrame format provided by the Python library pandas.

[0081] According to one embodiment, after the data analysis device (371) obtains a file from cloud storage (343), it may delete the temporary file stored in the cloud storage and drop the temporary table in the data warehouse. In standard SQL, 'drop' is used as a command to remove a table from a database, and 'delete' is used as a command to remove the contents of a table while maintaining the table. In the present disclosure, 'drop' may be used as a term to remove the table itself from a database or dataset. Meanwhile, since it is uncommon for the operation of deleting part or all of the data of a temporary table in a data warehouse to be performed, the expression 'delete a table' in a data warehouse may mean 'drop a table'.

[0082] FIG. 4 is a flowchart for explaining a data analysis method according to one embodiment of the present disclosure.

[0083] According to one embodiment, in operation 411, a data analysis device (e.g., the data analysis device (255) of FIG. 2 or the data analysis device (371) of FIG. 3) may create a temporary table to store data in order to extract data contained in a data warehouse (e.g., the data warehouse (251) of FIG. 2 or the data warehouse (341) of FIG. 3). In operation 413, a DB query may be written to obtain data to be analyzed, and the result of executing the DB query may be stored in the created temporary table.

[0084] Table 1 shows the pseudocode of exemplary SQL statements that can perform the temporary table creation operation (operation 411) and the DB query result saving operation (operation 413).

[0085]

[0086] Table 2 shows other exemplary SQL statements in pseudocode that can perform the temporary table creation operation (operation 411) and the DB query result saving operation (operation 413).

[0087]

[0088] According to one embodiment, after creating a temporary table as shown in Table 1, data can be stored in the temporary table through a database (DB) query statement, and an SQL statement capable of simultaneously performing table creation and DB query result storage as shown in Table 2 is also possible. However, even in the case where a single create table statement is executed as shown in Table 2 to create a temporary table and store information of the temporary table from another table, the internal execution procedure may be performed according to the process of creating a table, the process of extracting data from another table, and the process of inputting the extracted data into the created table.

[0089] According to one embodiment, the above-described SQL statement can be entered directly in a command input such as a cloud console, and the SQL statement can also be executed through program code such as Python.

[0090] For example, Google BigQuery creates a unique project within an account, and a project can contain one or more datasets. A dataset corresponds to a database in a relational database management system and can contain one or more tables. When extracting data from a table within the same project and moving it to a temporary table, the process can be completed quickly. Since the nature of data warehouses may prohibit deleting, modifying, or adding contents to original tables, the work can be performed by extracting necessary data from the original tables using SQL statements and creating a temporary table.

[0091] According to one embodiment, when a temporary table is created, meta information related to the temporary table, such as the schema of the temporary table, the number of rows of the temporary table, and the current resource status of the data analysis device, can be obtained (operation 425). Based on the meta information, the resources required by the data analysis device can be calculated (operation 426).

[0092] According to one embodiment, based on the amount of resources calculated, it can be determined whether there are sufficient resources for data analysis (operation 427), and based on whether there are sufficient resources for data analysis, the excessively allocated resources can be returned. Data stored in a temporary table can be transferred to cloud storage (operation 442) and saved as a temporary file.

[0093] According to one embodiment, based on the fact that there are insufficient resources for data analysis, it can be determined whether resources can be allocated to a data analysis device (operation 428), and based on the fact that the resource allocation device can allocate resources to the data analysis device, additional resources can be allocated to the data analysis device (operation 429). After resources are allocated to the data analysis device, files can be transferred to cloud storage (operation 442).

[0094] According to one embodiment, the actual data size and the required memory size do not match, making it difficult to select appropriate resources for data analysis. For example, if data analysis is performed with insufficient resources allocated, the analysis may not be easy, and if too many resources are allocated, inefficiency in resource allocation may occur.

[0095] According to one embodiment, resource allocation can be determined (operation 429) by using at least one of a predefined rule-based resource allocation method and a deep learning training data-based resource allocation method, or by combining the two methods.

[0096] According to one embodiment, a resource allocation method based on predefined rules can estimate the required memory using the number of rows in a temporary table and the schema of the temporary table. Based on the memory already allocated, the amount of additional memory required for allocation can be calculated (operation 426).

[0097] According to one embodiment, a deep learning-based resource allocation method can estimate the optimal memory required for data analysis by applying a deep learning model based on meta-information such as the number of rows in a temporary table, the number of columns in a temporary table, and the schema of a temporary table, and the amount of memory used for actual data analysis. Based on the already allocated memory, the amount of additional memory required for allocation can be calculated (operation 426).

[0098] According to one embodiment, a suitable processor (CPU, GPU, NPU, TPU) can be selected based on the calculated memory, and the number of processors required can be determined. By considering the algorithms, processes, and logic used for data analysis, a processor suitable for allocation as a resource can be determined.

[0099] According to one embodiment, resources can be allocated to a data analysis device (operation 429) based on at least one of the calculated amount of memory, meta-information, deep learning training results, algorithms used for data analysis, processes, and logic. If the data analysis device lacks the necessary resources, additional resources can be allocated, and if an excessive amount of resources has already been allocated, the allocated resources can be recovered. If resource allocation fails, the operation can be stopped and the procedure can be terminated.

[0100] According to one embodiment, the free operation of a data warehouse may have a very slow speed for transmitting data externally. However, even when using the free operation of the data warehouse, cloud storage provided by a cloud service including the data warehouse can receive data from the data warehouse very quickly. When attempting to acquire data using the free operation of the data warehouse, downloading via cloud storage can improve the download speed. The usage fee for the cloud storage download service is lower than that of the data warehouse, and costs can be further reduced by using methods such as splitting and compression.

[0101] According to one embodiment, instead of directly downloading data from a data warehouse, if the data is transferred from cloud storage and then downloaded from cloud storage, the cost can be reduced and the download time saved.

[0102] According to one embodiment, data stored in a temporary table can be transferred to cloud storage (operation 442), and the transferred data can be stored as a temporary file in cloud storage.

[0103] According to one embodiment, when storing a temporary file in cloud storage, the file may be compressed to reduce storage space, the file may be split and stored, and the file may be compressed and stored after splitting.

[0104] According to one embodiment, the data analysis device may obtain a data file by downloading a temporary file stored in cloud storage (operation 454). The data file may be downloaded as an actual file, or the data analysis device may not save it as an actual file but instead convert the data of the file into a data format for analysis (operation 460) and load it into memory. The data analysis device may analyze the converted data loaded into memory (operation 490).

[0105] According to one embodiment, when a file stored in cloud storage is split / compressed, multiple compressed files can be downloaded and decompressed, and the decompressed file can be converted into a format suitable for data analysis in a programming language such as Python, RUBY, Perl, PHP, Java, C / C++ (Operation 460). For example, the file can be converted into a dataFrame of the Python library pandas.

[0106] According to one embodiment, when downloading and decompressing split / compressed files, data can be converted (operation 460) by processing in memory without creating a physical file.

[0107] According to one embodiment, after obtaining the transformed data, the temporary files in the cloud storage can be deleted (operation 481). After obtaining the transformed data, the temporary table in the data warehouse can be dropped (using the drop table statement) (operation 481). Meanwhile, after the temporary file is stored in the cloud storage, the temporary table in the data warehouse can be dropped (operation 481).

[0108] FIG. 5 is a flowchart illustrating a data analysis method according to one embodiment of the present disclosure.

[0109] The data analysis device according to the present disclosure acquires original data to be analyzed and performs data analysis.

[0110] According to one embodiment, the data analysis device may establish a connection with a data warehouse (operation 510) to obtain original data. In order for the data analysis device to perform a task in the data warehouse, it may be possible to transmit a task execution request message, and for the data warehouse to receive the request message, perform the task. Meanwhile, in order for the data analysis device to perform a task in the data warehouse, the data analysis device may execute a command to perform the task in the data warehouse while connected to the data warehouse.

[0111] According to one embodiment, operations in the data warehouse can be performed by executing SQL statements, commands, utilities, and programming code while connected to the data warehouse through a service or API provided by the data warehouse. According to one example, a temporary table can be created (operation 511), original data can be stored in the temporary table (operation 513), and original data can be retrieved (operation 555).

[0112] According to one embodiment, a programming language such as Python, RUBY, Java, or PHP provided by the data analysis device can connect to the data warehouse and perform operations in the data warehouse using APIs provided by the data warehouse. According to one example, the data analysis device can create a temporary table (operation 511), store original data in the temporary table (operation 513), and retrieve original data (operation 555) through a connection with the data warehouse. Operations 511 and 513 can be performed while the data analysis device is connected to the data warehouse using standard SQL disclosed in Table 1 or Table 2. Alternatively, operations 511 and 513 can be performed by a method in which the data analysis device requests the execution of standard SQL statements in the data warehouse using a program such as Python. Meanwhile, given that instructions for operations to be performed in the data warehouse are given by the data analysis device, it may be interpreted as an operation requesting the said operation.

[0113] According to one embodiment, the data warehouse is provided as an OLAP (onLine analytical processing) service and may have only read rights to the information in the table and no write rights, but the original data can be stored (operation 513) by creating a temporary table in the data warehouse and obtaining metadata related to the original data.

[0114] According to one embodiment, the data analysis device may directly acquire original data from a data warehouse or acquire original data stored in a temporary table (operation 555). The original data may be acquired in various data structures such as SQL, JSON, and DB dumps, and may be acquired in the form of a file or a stream. Meanwhile, when the data analysis device acquires data from a data warehouse, it may select a free operation or a paid operation.

[0115] Table 3 is a table comparing the time and cost required for free and paid tasks based on the size of the downloaded data.

[0116]

[0117] Referring to Table 3, for free operations, it takes approximately 4 hours to acquire 7.6GB. On the other hand, for paid operations, it is confirmed that processing is completed within 5 minutes, and costs are incurred. Costs are incurred based on the size of the processed bytes, which is the size of the original data. It can be seen that the processed bytes differ from the size of the downloaded data.

[0118] According to one embodiment, in the case of paid operations, the cost is determined based on the amount of processed bytes rather than the size of the downloaded data, and free operations have the problem of not being suitable for processing big data due to slow transmission speeds, and in the case of paid operations, for data of millions of terabytes or more, the cost burden for big data analysis may be high.

[0119] According to one embodiment, when a data analysis device acquires original data, it can convert the data into a format suitable for data analysis (operation 560). The acquired original data can be saved to a file, or instead of saving it to a file, it can be loaded into memory and converted into a format suitable for data analysis (operation 560) the original data loaded into memory. Data analysis (operation 590) can be performed using the data converted into a format suitable for data analysis. For example, the data can be converted into a pandas dataFrame format supported by Python, and data analysis can be performed using a library supported by Python.

[0120] FIG. 6 is a flowchart illustrating a data analysis method according to one embodiment of the present disclosure.

[0121] Referring to FIG. 6, a data analysis device according to the present disclosure acquires original data to be analyzed and performs data analysis. FIG. 6 performs the same operation as FIG. 5, except for operation 620 (operations 621 to 629). Specifically, operations 510, 511, 513, 555, 560, and 590 correspond to operations 610, 611, 613, 655, 660, and 690, respectively. The following description focuses on operation 620.

[0122] The data analysis device according to the present disclosure may be provided as a cloud service. Since the data analysis device may require a large amount of processor resources, such as CPUs, GPUs, NPUs, and TPUs, as well as memory resources, to process big data, the data analysis device may perform data analysis by allocating resources provided by the cloud service. Recently, cloud services can adaptively provide an environment capable of processing big data by dynamically allocating resources.

[0123] According to one embodiment, an adaptive data analysis device can be configured by dynamically allocating resources to the data analysis device. Even when using free operations, the time required for data acquisition can be shortened, and costs incurred due to prolonged occupation in the data analysis environment can be reduced by allocating the resources of the data analysis device.

[0124] According to one embodiment, in order for a data analysis device to perform data analysis, it must secure resources necessary for data analysis. The amount of resources required for data analysis is generally closely related to the size of the original data. The original data may be stored in a data warehouse provided as a cloud service. When the original data is stored in a temporary table in the data warehouse, metadata related to the original data may be generated (operation 621). The data analysis device may obtain metadata such as the size of the original data and the table schema from the data warehouse (operation 625) and calculate the amount of resources required for data analysis.

[0125] According to one embodiment, the data type of each column constituting the table can be determined from the table schema. For example, int64 is a 64-bit integer type with a size of 8 bytes, and if the column is in string format, the size defined for strings in the data warehouse can be the size of the corresponding column. The sizes of the columns can be summed to calculate the size of each row, and this can be used to calculate the required resources by multiplying it by the total number of rows.

[0126] Mathematical formula 1 is a formula for calculating the amount of required memory.

[0127]

[0128] According to one embodiment, if the table consists of m columns, the size of each column can be obtained from the table schema. The amount of required memory can be calculated by considering weights on the value obtained by multiplying the sum of the sizes of all columns by the number of rows.

[0129] According to one embodiment, when multiple tables are required for data analysis, the required memory can be calculated by summing the required memories calculated for each table according to Equation 1.

[0130] According to one embodiment, since the memory required for data analysis typically requires a larger size than the original data, the weight w1 typically has a value of 1 or greater. Additionally, since it is desirable to minimize the size of the original data by excluding unnecessary data so that the weight w1 does not become 1.0 or less, the objective of the invention can be achieved even if the case is limited to a weight w1 of 1.0 or greater. However, there may also be cases where the weight w1 of Equation 1 is 1 or less. For example, in order to quickly extract the original data, if the entire data is acquired as original data and a file containing only some of the columns of the original data is copied to cloud storage, the weight w1 may be 1.0 or less. The weight w1 can be determined empirically through trial and error based on various data analyses.

[0131] Mathematical formula 2 is another formula for calculating the amount of memory required.

[0132]

[0133] According to one embodiment, when data analysis is performed using original data, the data analysis method is often already determined. Therefore, the size of the analysis result data can be predicted after the data analysis. For data analysis, memory for the data to be analyzed is required, and since the analysis result data is closely related to the data to be analyzed, the analysis result data may include the data to be analyzed. For example, the analysis result data may have the same or more rows than the data to be analyzed. The analysis result data may include more columns than the data to be analyzed, and the added columns may include analysis results derived from the data to be analyzed.

[0134] According to one embodiment, the actual bytes of the table to be analyzed are related to a value obtained by summing the sizes of the data actually used when the original data is stored in a temporary table, and the logical bytes of the table to be analyzed can be determined based on the row size and number of rows of the temporary table. For example, the logical bytes of the table to be analyzed may be the value obtained by multiplying the row size and the number of rows of the temporary table. Since the row size of the table to be analyzed is the value obtained by summing the data sizes of the columns, the row size of the table to be analyzed can be easily obtained from the table schema information.

[0135] According to one embodiment, the logical bytes of the table under analysis are generally larger than the actual bytes of the table under analysis. For example, if the data type is a variable-length string, the data size defining the column is the maximum size of the data to be stored, so the actual size used is smaller than the data size defining the column. However, if the actual stored data is fully stored up to the maximum size allowed for each column, the actual bytes of the table under analysis may be larger than the logical bytes of the table under analysis. Since the actual bytes of the table under analysis have a data structure for handling variable lengths, additional data structure information may be included.

[0136] According to one embodiment, it can be empirically confirmed that allocating required memory in proportion to the size of the analysis result data can be an appropriate memory allocation rule. However, if there are many empty values ​​in a specific column of the data to be analyzed, the memory required to analyze the data may be small. Reflecting this, Equation 2 was derived, and although the calculation formula is simple, it can be a powerful resource allocation rule. Weight w2 can be interpreted in a similar sense to weight w1 related to Equation 1, and can typically be a value greater than or equal to 1 (e.g., 1.5), but can also be less than or equal to 1.

[0137] According to one embodiment, the value that is not smaller among the request memory 1 of mathematical formula 1 and the request memory 2 of mathematical formula 2 can be determined as the request memory.

[0138] According to one embodiment, a data warehouse often adopts a database in a column-based format, and the total size of a specific column can be easily calculated, and the average size of specific columns can be determined using the total number of rows. Since the amount of data used for data analysis is influenced by the average size of each column, the amount of required memory can be calculated using the average size of each column.

[0139] According to one embodiment, a deep learning-based resource allocation method can calculate the optimal resources to be allocated through learning along with meta-information. By obtaining meta-information such as a table schema, the number of rows in each table, and an algorithm used for data analysis, and obtaining the amount of memory required for actual data analysis, the required memory can be calculated according to the learned result.

[0140] According to one embodiment, the memory required for data analysis can be estimated by analyzing the programming code used for data analysis through LLM. With the recent advancement of generative AI technology, the required memory can be calculated based on the programming code. Since the analysis algorithm is determined according to the data analysis method and metadata of the data to be analyzed can be obtained, the resources required for data analysis can be determined using LLM. It is also possible to determine the type of processor (CPU, GPU, NPU, TPU) that is more suitable according to the data analysis algorithm. When the required resources are calculated according to simple rules, the data analysis device can calculate the required resources. However, when calculating the required resources according to the data analysis method using LLM, or calculating resources through deep learning, it would be desirable to configure a separate resource calculation device.

[0141] According to one embodiment, the data analysis device may request resource allocation from the resource allocation device based on meta-information (operation 627) and receive resource allocation from the resource allocation device (operation 629). Since there are not enough resources to allocate to the data analysis device, the data analysis device may not receive sufficient resources. In this case, data analysis may be impossible, so the procedure for data analysis may be terminated. On the other hand, even if data analysis is impossible, the data to be analyzed may be secured as a file, and data analysis may be performed later when resource allocation becomes possible.

[0142] The present disclosure can provide a data analysis method that guarantees speeds comparable to paid operations while incurring no or minimal additional costs, such as those associated with free operations.

[0143] FIG. 7 is a flowchart illustrating a data analysis method according to one embodiment of the present disclosure.

[0144] According to one embodiment, the data analysis device may establish a first connection with a data warehouse to obtain original data (operation 710). While connected to the data warehouse, the data analysis device may create a temporary table in the data warehouse (operation 711) and execute a DB query statement to create an original target to be analyzed, thereby storing data (query result) in the temporary table (operation 713).

[0145] According to one embodiment, metadata related to a temporary table in which original data is stored in a data warehouse may be generated (operation 721), and the data analysis device may acquire the metadata (operation 725), calculate the amount of resources to be used for subsequent data analysis (operation 790), request resource allocation from the resource allocation device (operation 727), and receive the corresponding resources from the resource allocation device (operation 729). If resource allocation fails, the process may be terminated (see operation 428 in FIG. 4), or the process may be terminated after the data analysis device downloads the data to be analyzed (operation 754). If resource allocation fails, data analysis (operation 790) is impossible, but data file acquisition (operation 754) may still be possible; therefore, after acquiring the data file, data analysis can be performed once resource allocation becomes possible.

[0146] According to one embodiment, the operation of allocating resources based on meta-information (operation 720) may be performed after the data conversion operation (operation 760) is performed and before the data analysis operation (operation 790) is performed. Once meta-information is generated, the possibility of data analysis can be determined based on whether resource allocation is possible, and if data analysis is not possible, the procedure can be terminated to prevent unnecessary work. However, since the data to be analyzed may be acquired first and data analysis may be performed later (when resources for data analysis can be secured), operation 720 may be performed between operation 760 and operation 790.

[0147] According to one embodiment, the operation 720 of allocating resources based on meta-information can be performed similarly to the operation 620 of FIG. 6.

[0148] According to one embodiment, the original data to be analyzed can be stored in a temporary table of a data warehouse (Operation 713), and then transferred to cloud storage in the form of a file (Operation 742) and stored (Operation 744). The data analysis device can establish a first connection with the data warehouse and then establish a second connection with cloud storage based on the first connection (Operation 740). That is, the data analysis device can connect to cloud storage via the data warehouse and can use the functions of cloud storage supported by the data warehouse. The original data in the temporary table of the data warehouse can be converted into a file and transferred to cloud storage (Operation 742).

[0149] For example, BigQuery, a data warehouse, can (1) save information about BigQuery tables to Google Cloud Storage as a file using the Google Cloud Console, (2) save information about BigQuery tables to Google Cloud Storage as a file using gsutil, and (3) save information about BigQuery tables to Google Cloud Storage as a file using Python programming. The methods described above are exemplary and are not intended to limit the methods of using cloud storage functions in a data warehouse in the present disclosure.

[0150] According to one embodiment, the data analysis device can store information of a temporary table in the form of a file in cloud storage while connected to a data warehouse (operation 744). Depending on the data to be stored, it may be stored by splitting or compressing.

[0151] According to one embodiment, when a file is created in cloud storage, the data analysis device can directly connect to the cloud storage (operation 750) and acquire the file (operation 754). The acquired file can be converted into a data structure that facilitates data analysis (operation 760), and data analysis (790) can be performed based on the converted data.

[0152] FIG. 8 is a flowchart illustrating a data analysis method according to various embodiments of the present disclosure.

[0153] Since the operations 810 to 860 disclosed in FIG. 8 are similar to the operations 710 to 760 of FIG. 7, reference may be made to the descriptions in FIG. 5, FIG. 6 and FIG. 7. Specifically, operations 810, 811, 813, 820, 821, 825, 827, 829, 840, 842, 844, 850, 854, and 860 of FIG. 8 may correspond to operations 710, 711, 713, 720, 721, 725, 727, 729, 740, 742, 744, 750, 754, and 760 of FIG. 7, respectively.

[0154] According to one embodiment, after converting the data file obtained in operation 860, the data analysis device may request the hash value of the temporary file from the cloud storage (operation 871). The data analysis device receives the hash value of the temporary file from the cloud storage (operation 873), and the data analysis device may calculate a hash value from the obtained data file and compare the hash value of the temporary file received from the cloud storage with the hash value calculated from the data file (operation 875). Based on the result of the comparison that the two hash values ​​match each other, it can be confirmed that the file download from the cloud storage was successfully performed. The hash value calculation may be performed by the data analysis device and the cloud storage using the same method, and at least one of the hash algorithms md5, sha-1, sha-2, sha-3, BLAKE2, and BLAKE3 may be used.

[0155] According to one embodiment, operations (operations 871, 873, and 875) for confirming that a file download from cloud storage has been successfully performed using the hash value described above may be performed between operation 854 (data file acquisition operation) and operation 860 (data conversion operation), unlike as illustrated in FIG. 8.

[0156] According to one embodiment, if the hash value of a temporary file received from cloud storage and the hash value calculated from a data file are compared (operation 875) and do not match, the previously downloaded data file can be deleted and the data file containing the temporary file information can be retrieved again from cloud storage.

[0157] According to one embodiment, after acquiring a file from cloud storage (operation 854), the data analysis device may request the data warehouse to drop the temporary table (operation 881) and request the cloud storage to delete the temporary file (operation 883). Since the data analysis device is connected to the data warehouse through a first connection, it may request the drop of the temporary table in the data warehouse through an SQL statement to drop the temporary table. The method of requesting the drop may be (1) to connect to a console and execute the SQL statement, (2) to execute the SQL statement through a utility, or (3) to execute the SQL statement through programming such as Python, Ruby, Java, C / C++, or C#. When executing the SQL statement through the aforementioned utility, for batch execution, a file containing an execution list as a utility command may be executed. When using a programming language such as Python, programming may be done to operate the above-mentioned function, and the SQL statement may be executed by executing Python code. When a request for temporary table drop is made by the method described above, the temporary table will be dropped (not shown) from the data warehouse. The data analysis device is connected to cloud storage through a third connection and can request temporary file deletion through the third connection. In one example, the method for requesting temporary file deletion can be (1) requesting file deletion through a service provided by cloud storage, (2) requesting file deletion through a utility, and (3) requesting file deletion through programming such as Python, Ruby, Java, C / C++, or C#. When requesting file deletion through the utility described above, for batch execution, a file listed in the execution list as a utility command can be executed.When using a programming language such as Python, you can program to operate the aforementioned function and request file deletion by executing Python code. When a request for temporary file deletion is made by the aforementioned method, the temporary file will be deleted from the cloud storage (not shown). Since both the data warehouse and cloud storage use usage-based pricing, after acquiring data for data analysis, you can delete the temporary table in the data warehouse and the temporary file in the cloud storage to avoid being charged.

[0158] According to one embodiment, the data analysis device may acquire a data file from cloud storage (operation 854). The data analysis device may convert the acquired file into a data structure that facilitates data analysis (operation 860) and perform data analysis (890) based on the converted data.

[0159] Table 4 is a table comparing the download time required for paid operations and the method proposed in this disclosure according to the size of the downloaded data.

[0160]

[0161] According to one embodiment, it can be seen that the data acquisition method can acquire data without additional cost while requiring a download time similar to that of a paid operation.

[0162] According to one embodiment, the data warehouse is a cloud service for data processing and is optimized for data processing, and the cloud storage is a cloud service for storing and accessing files and is optimized for file downloading and uploading. By performing work centered on data processing in the data warehouse, moving the corresponding data from the cloud storage belonging to the same cloud service, and downloading files directly from the cloud storage, a data analysis process suitable for a distributed system can be established, and a cost-effective process can be established by performing data analysis in an environment optimized for data analysis.

[0163] In a data analysis device according to the present disclosure, when the at least one instruction is executed individually or collectively by the at least one processor, the data analysis device may cause the temporary table to be dropped through the first connection after a temporary file is stored in the cloud storage.

[0164] In a data analysis device according to the present disclosure, the at least one instruction, when executed individually or collectively by the at least one processor, may cause the data analysis device to delete the temporary file from the cloud storage through the third connection after the data analysis device has acquired the file from the cloud storage.

[0165] In a data analysis device according to the present disclosure, the at least one instruction, when executed individually or collectively by the at least one processor, may cause the data analysis device to receive a first hash code of the temporary file from the cloud storage, generate a second hash code based on the file, and determine that the first data file is the same as the temporary file based on the fact that the first hash code is the same as the second hash code.

[0166] In a data analysis device according to the present disclosure, the at least one instruction, when executed individually or collectively by the at least one processor, may cause the data analysis device to: receive a first hash code of the temporary file from the cloud storage; generate a second hash code based on the first data file; determine that the first data file is not identical to the temporary file based on the determination that the first hash code is not identical to the second hash code; and obtain a second data file containing information regarding the temporary file received again from the cloud storage based on the determination that the first data file is not identical to the temporary file.

[0167] In a data analysis device according to the present disclosure, the at least one instruction, when executed individually or collectively by the at least one processor, may cause the data analysis device to: obtain meta-information related to data stored in the temporary table through the first connection.

[0168] In a data analysis device according to the present disclosure, the meta-information may include at least one of the number of rows of a temporary table, the size of columns of a temporary table, the logical bytes of a temporary table, the actual bytes of a temporary table, and the size of the analysis result data. The logical bytes of the temporary table may be the value obtained by multiplying the number of columns of the temporary table by the sum of the column sizes of the temporary table. The actual bytes of the temporary table may be related to the size used by the temporary table.

[0169] In a data analysis device according to the present disclosure, when the at least one instruction is executed individually or collectively by the at least one processor, the data analysis device may cause: to calculate resources required for data analysis based on the meta-information and to obtain resources required for data analysis from a resource allocation device.

[0170] In a data analysis device according to the present disclosure, when the at least one instruction is executed individually or collectively by the at least one processor, the data analysis device causes the data analysis device to: calculate the resources required for data analysis based on the meta-information and obtain the resources required for data analysis from a resource allocation device, wherein the resources required for data analysis may be calculated by applying a weight to the value obtained by multiplying the sum of the sizes of the columns of the temporary table by the number of rows of the temporary table.

[0171] In a data analysis device according to the present disclosure, when the at least one instruction is executed individually or collectively by the at least one processor, the data analysis device is caused to: calculate the resources required for data analysis based on the meta-information and obtain the resources required for data analysis from a resource allocation device, wherein the resources required for data analysis may be calculated by applying a weight to the value obtained by multiplying the size of the analysis result data by the ratio of the logical bytes of the temporary table to the actual bytes of the temporary table.

[0172] A data analysis method according to the present disclosure may include: connecting to a data warehouse; creating a temporary table in the data warehouse; storing first data obtained from at least one table of the data warehouse in the temporary table; connecting to a cloud storage through the data warehouse; storing a temporary file containing information of the temporary table in the cloud storage; connecting to the cloud storage from the data analysis device; obtaining a first data file from the cloud storage; converting the first data file into second data; and analyzing the second data.

[0173] The data analysis method according to the present disclosure may further include the operation of dropping the temporary table through the first connection after the temporary file is stored in the cloud storage.

[0174] The data analysis method according to the present disclosure may further include the operation of deleting the temporary file from the cloud storage through the third connection after the data analysis device has acquired the first data file from the cloud storage.

[0175] A data analysis method according to the present disclosure may further include: receiving a first hash code of the temporary file from the cloud storage; generating a second hash code based on the first data file; and determining whether the first data file is identical to the temporary file based on the identity of the first hash code and the second hash code.

[0176] The data analysis method according to the present disclosure may further include an operation of determining that the first data file is the same as the temporary file based on the fact that the first hash code is the same as the second hash code.

[0177] The data analysis method according to the present disclosure may further include the operation of receiving the temporary file again from the cloud storage and obtaining the second data file based on the determination that the temporary file and the first data file are not identical to each other.

[0178] The data analysis method according to the present disclosure may further include the operation of obtaining meta-information related to the first data stored in the temporary table through the first connection.

[0179] In the data analysis method according to the present disclosure, the meta information may include at least one of the number of rows of a temporary table, the size of columns of a temporary table, the logical bytes of a temporary table, the actual bytes of a temporary table, and the size of the analysis result data.

[0180] In the data analysis method according to the present disclosure, the logical byte of the temporary table is the product of the number of columns of the temporary table and the sum of the column sizes of the temporary table, and the actual byte of the temporary table may be related to the size used by the temporary table.

[0181] The data analysis method according to the present disclosure may further include: an operation of calculating resources required for data analysis based on the meta-information; and an operation of obtaining resources required for data analysis from a resource allocation device.

[0182] The data analysis method according to the present disclosure further comprises: an operation of calculating resources required for data analysis based on the meta-information; and an operation of obtaining resources required for data analysis from a resource allocation device, wherein the resources required for data analysis may be calculated by applying a weight to a value obtained by multiplying the sum of the sizes of the columns of the temporary table by the number of rows of the temporary table.

[0183] The data analysis method according to the present disclosure further comprises: an operation of calculating resources required for data analysis based on the meta-information; and an operation of obtaining resources required for data analysis from a resource allocation device, wherein the resources required for data analysis may be calculated by applying a weight to a value obtained by multiplying the size of the analysis result data by the ratio of the logical bytes of the temporary table to the actual bytes of the temporary table.

[0184] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0185] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0186] Various embodiments of this document may be implemented as software (e.g., a program) comprising one or more instructions stored in a storage medium (e.g., internal memory or external memory) readable by a machine (e.g., an electronic device). For example, a processor of the machine (e.g., an electronic device) may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, "non-transitory" simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily in the storage medium.

[0187] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0188] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In a data analysis device, Communication circuit; Memory for storing at least one instruction; and It includes at least one processor comprising a processing circuit, When the above at least one instruction is executed individually or collectively by the above at least one processor, the data analysis device: Through a first connection established between the data analysis device and the data warehouse, a temporary table is created in the data warehouse, and Through the first connection above, first data obtained from at least one table of the data warehouse is stored in the temporary table, and Through a second connection established between the data warehouse and the cloud storage, a temporary file containing information of the temporary table is stored in the cloud storage, and The data analysis device obtains a first data file by receiving the temporary file from the cloud storage through a third connection connected to the cloud storage, and Convert the above first data file into second data, and causing the analysis of the above second data Data analysis device.

2. In Paragraph 1, When the above at least one instruction is executed individually or collectively by the above at least one processor, the data analysis device: After a temporary file is stored in the cloud storage, causing the temporary table to be dropped through the first connection Data analysis device.

3. In Paragraph 1 or 2, When the above at least one instruction is executed individually or collectively by the above at least one processor, the data analysis device: After the data analysis device acquires the first data file from the cloud storage, it causes the temporary file to be deleted from the cloud storage through the third connection. Data analysis device.

4. In any one of paragraphs 1 through 3, When the above at least one instruction is executed individually or collectively by the above at least one processor, the data analysis device: Receive the first hash code of the above temporary file from the cloud storage, and A second hash code is generated based on the first data file above, and Causing to determine that the first data file is the same as the temporary file based on the fact that the first hash code is the same as the second hash code Data analysis device.

5. In any one of paragraphs 1 through 4, When the above at least one instruction is executed individually or collectively by the above at least one processor, the data analysis device: Receive the first hash code of the above temporary file from the cloud storage, and A second hash code is generated based on the first data file above, and Based on the confirmation that the first hash code is not identical to the second hash code, it is determined that the first data file is not identical to the temporary file, and Based on the determination that the first data file is not identical to the temporary file, the temporary file is received again from the cloud storage to obtain the second data file, and Causing the above second data file to be converted into the above second data Data analysis device.

6. In any one of paragraphs 1 through 5, When the above at least one instruction is executed individually or collectively by the above at least one processor, the data analysis device: Causing to obtain meta-information related to the first data stored in the temporary table through the first connection above Data analysis device.

7. In Paragraph 6, The above meta-information includes at least one of the number of rows in the temporary table, the size of the columns in the temporary table, the logical bytes of the temporary table, the actual bytes of the temporary table, and the size of the analysis result data. The logical byte of the above temporary table is the value obtained by multiplying the number of columns of the above temporary table by the sum of the column sizes of the above temporary table, and The actual bytes of the above temporary table are related to the size used by the above temporary table. Data analysis device.

8. In Paragraph 6 or 7, When the above at least one instruction is executed individually or collectively by the above at least one processor, the data analysis device: Based on the above meta-information, the resources required for data analysis are calculated, and Causing to obtain resources necessary for the data analysis from the resource allocation device Data analysis device.

9. In Paragraph 7, When the above at least one instruction is executed individually or collectively by the above at least one processor, the data analysis device: Based on the above meta-information, the resources required for data analysis are calculated, and Causing to obtain resources necessary for the data analysis from the resource allocation device, The resources required for the above data analysis are calculated by applying weights to the value obtained by multiplying the sum of the sizes of the columns of the above temporary table by the number of rows of the above temporary table. Data analysis device.

10. In Paragraph 7, When the above at least one instruction is executed individually or collectively by the above at least one processor, the data analysis device: Based on the above meta-information, the resources required for data analysis are calculated, and Causing to obtain resources necessary for the data analysis from the resource allocation device, The resources required for the above data analysis are calculated by applying weights to the value obtained by multiplying the size of the analysis result data by the ratio of the logical bytes of the temporary table to the actual bytes of the temporary table. Data analysis device.

11. Regarding data analysis methods, The operation of a data analysis device connecting to a data warehouse; The operation of creating a temporary table in the above data warehouse; The operation of storing first data obtained from at least one table of the data warehouse in the above temporary table; The operation of connecting to cloud storage through the above data warehouse; The operation of storing a temporary file containing information of the temporary table in the cloud storage; The operation of connecting to the cloud storage from the above data analysis device; The operation of receiving the temporary file from the cloud storage and obtaining the first data file; The operation of converting the above-mentioned first data file into second data; and A data analysis method including the operation of analyzing the above second data.

12. In Paragraph 11, After the temporary file is stored in the cloud storage, the operation of dropping the temporary table is further included. Data analysis methods.

13. In Paragraph 11 or 12, The above data analysis device further includes the operation of deleting the temporary file from the cloud storage after acquiring the first data file from the cloud storage. Data analysis methods.

14. In any one of paragraphs 11 through 13, The operation of receiving the first hash code of the above temporary file from the cloud storage; The operation of generating a second hash code based on the first data file; and The method further includes an operation of determining that the first data file is the same as the temporary file based on the fact that the first hash code is the same as the second hash code. Data analysis methods.

15. A computer-readable medium storing a program for executing a data analysis method, wherein the program is used for at least one operation of the data analysis device when executed by at least one processor of the data analysis device, and The above at least one operation is: The operation of a data analysis device connecting to a data warehouse; The operation of creating a temporary table in the above data warehouse; The operation of storing first data obtained from at least one table of the data warehouse in the above temporary table; The operation of connecting to cloud storage through the above data warehouse; The operation of storing a temporary file containing information of the temporary table in the cloud storage; The operation of connecting to the cloud storage from the above data analysis device; The operation of receiving the temporary file from the cloud storage and obtaining the first data file; The operation of converting the above-mentioned first data file into second data; and A computer-readable recording medium comprising an operation to analyze the above-mentioned second data.

Images