Data screening program, data screening method, and information processing device

Abstract

The present invention improves efficiency of data extraction.　A processing unit (12) receives a designated number of pieces of data to be extracted from among a plurality of pieces of data each indicating a point in an N-dimensional space (20) (where N is an integer of 1 or greater). The processing unit (12) divides the N-dimensional space (20) into a first number of areas. For each of the first number of areas, the processing unit (12) performs a selection process for selecting one piece of data out of pieces of data from among the plurality of pieces of data which are present in the area. The processing unit (12) determines whether or not the total number of pieces of selected data is the same as the designated number. When the total number is not the same as the designated number, the processing unit (12) changes the number of areas partitioning the N-dimensional space (20) to a second number different from the first number, and performs the selection process.

Description

Data screening program, data screening method, and information processing apparatus

[0001] The present invention relates to a data screening program, a data screening method, and an information processing apparatus.

[0002] Data screening may be performed by an information processing apparatus. Data screening is a technique for selecting data from a huge data group and reducing the total number of data. For example, data screening may improve the accuracy of data by deleting missing values and outliers, or may improve the processing speed for 3D point cloud data of a 3D model.

[0003] Here, based on the information of the point cloud obtained by measuring the real space, the region containing the point cloud is divided into a plurality of regions by the Bounding Volume Hierarchy (BVH) method, and the point cloud is thinned out at a certain ratio for each region according to the number and density of the point cloud contained in each divided region. There is a proposal for a system.

[0004] Also, in a system that forms an integrated 3D model of a scene by combining a plurality of data of point clouds obtained from a plurality of scanners such as LiDAR (Light Detection And Ranging), a method for deleting redundant points in the overlapping region in each data has been proposed.

[0005] Japanese Patent Application Laid-Open No. 2017-54468, US Patent No. 10346998

[0006] In data screening, it is conceivable to reduce data while ensuring the diversity of data possessed by the original data group. For example, in the technology of artificial intelligence (AI: Artificial Intelligence), the diversity of training data used for machine learning affects the generalization performance of the machine learning model created based on the training data.

[0007] In this case, the method for extracting data from the original dataset becomes the problem. For example, one method could be to calculate the distance between each pair of data, extract one pair of data that is relatively close in distance, and discard the other. However, this method would require calculating the distance for every single pair of data in the dataset. Therefore, the larger the dataset being screened, the greater the processing cost required to extract data from it.

[0008] In one aspect, the present invention aims to improve the efficiency of data extraction.

[0009] In one embodiment, a data screening program is provided. This data screening program causes a computer to perform the following processes: The computer accepts a specified number of data to be extracted from a plurality of data, each representing a point in an N-dimensional space (where N is an integer greater than or equal to 1). The computer divides the N-dimensional space into a first number of areas. For each of the first number of areas, the computer performs a selection process to select one data from among the plurality of data present in that area. The computer determines whether the total number of selected data is equal to the specified number. If the total is not equal to the specified number, the computer changes the number of areas dividing the N-dimensional space to a second number different from the first number and performs the selection process again.

[0010] In one embodiment, a data screening method performed by a computer is provided. In another embodiment, an information processing device having a storage unit and a processing unit is provided.

[0011] In one aspect, data extraction can be made more efficient. The above and other objectives, features and advantages of the present invention will become apparent from the following description in conjunction with the accompanying drawings illustrating preferred embodiments as examples of the present invention.

[0012] This is a diagram illustrating the information processing device of the first embodiment. This is a diagram showing an example of the hardware of the information processing device of the second embodiment. This is a diagram showing an example of data screening. This is a diagram showing an example of the functions of the information processing device. This is a flowchart showing an example of data screening. This is a diagram showing an example of data screening. This is a flowchart showing a comparative example. This is a diagram showing an example of data screening results. This is a diagram showing an example of data screening results. This is a diagram illustrating the computational complexity of data screening. This is a diagram showing another comparative example. This is a diagram showing an example of data selection by the information processing device of the second embodiment. This is a flowchart showing a modified version of the first embodiment. This is a flowchart showing a modified version of the second embodiment.

[0013] Hereinafter, this embodiment will be described with reference to the drawings. [First Embodiment] The first embodiment will be described.

[0014] Figure 1 illustrates an information processing device according to a first embodiment. The information processing device 10 performs data screening on a dataset containing multiple data points. Each of the multiple data points represents a point in an N-dimensional space 20, where N is an integer greater than or equal to 1. In Figure 1, the N-dimensional space 20 is conveniently represented on a two-dimensional plane. The points represented by each of the multiple data points may also be points projected into the N-dimensional space by dimensionality reduction applied to each of the multiple original data points representing points in an M-dimensional space, where M is an integer greater than N. Existing methods such as principal component analysis and singular value decomposition can be used for dimensionality reduction. The information processing device 10 can use the dataset after data screening, for example, as training data in machine learning.

[0015] The information processing device 10 includes a storage unit 11 and a processing unit 12. The storage unit 11 may be a volatile semiconductor memory such as RAM (Random Access Memory), or a non-volatile storage such as an HDD (Hard Disk Drive) or flash memory.

[0016] The processing unit 12 is a processor such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), or DSP (Digital Signal Processor). However, the processing unit 12 may also include application-specific electronic circuits such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array). The processor executes programs stored in memory such as RAM (which may also be the storage unit 11). A collection of multiple processors is sometimes called a "multiprocessor" or simply a "processor."

[0017] The memory unit 11 stores multiple data points. These multiple data points correspond to the dataset to be screened. These multiple data points represent multiple points in the N-dimensional space 20 (step ST1). Figure 1 illustrates multiple points corresponding to multiple data points plotted in the N-dimensional space 20. Note that the N-dimensional space 20 is shown in two dimensions for convenience.

[0018] The processing unit 12 accepts a specified number of data to be extracted from multiple data sets. For example, the processing unit 12 accepts a specified number entered by the user into the information processing device 10. This specified number corresponds to the number of data to be retained as a result of data screening.

[0019] The processing unit 12 divides the N-dimensional space into multiple areas by dividing it into a first number of N-dimensional grids. The grids are N-dimensional hypercubes. For example, when N=1, the grids are line segments. When N=2, the grids are squares. When N=3, the grids are cubes. Even when N=4 or greater, hypercubes with the same side length can be defined. Also, all grids are the same size. The processing unit 12 divides the N-dimensional space into the same number of areas as the number of grids by arranging each grid adjacent to the others without gaps.

[0020] For example, the processing unit 12 divides the N-dimensional space 20 into four areas 21a, 22a, 23a, and 24a by dividing the N-dimensional space 20 into four grids 21, 22, 23, and 24. For each of the multiple areas, the processing unit 12 performs a selection process to select one data from among multiple data that exist in that area (step ST2). From step ST2 onward, points corresponding to the data selected by the selection process are shown as black circles, and points corresponding to data that are not selected are shown as dotted circles. Also, if no data belongs to a certain area, no data is selected from that area.

[0021] For example, each of the multiple data points may have an index. The index is a number that identifies the data, for example, 1, 2, 3, ... The processing unit 12 may select one data point from each area based on the index. In one example, the processing unit 12 may select the data point with the smallest index among the data points belonging to a certain area. Alternatively, the processing unit 12 may select the data point with the largest index, or the data point with the median index, among the data points belonging to a certain area.

[0022] In another example, the processing unit 12 may select data from among the data belonging to a certain area that corresponds to the point closest to the center of that area, or data that corresponds to the point closest to the centroid of each point belonging to that area. However, the processing unit 12 can reduce the cost of the selection process by selecting data using a simpler method, such as selection based on an index, compared to selecting data by calculating the centroid, etc.

[0023] The processing unit 12 determines whether the total number of data selected in the current selection process is the same as the specified number. If the total number is the same as the specified number, the processing unit 12 outputs information indicating the data selected in the current selection process as the extraction result of data screening. If the total number is not the same as the specified number, the processing unit 12 changes the number of grids dividing the N-dimensional space to a second number different from the first number and executes the selection process.

[0024] For example, suppose the number of specified items received is "8". The processing unit 12 determines that the total number of data items selected in the selection process of step ST2, "4", is different from the specified number "8". The processing unit 12 compares the total number "4" with the specified number "8" and detects that the total number "4" is smaller than the specified number "8". Then, the processing unit 12 changes the number of grids to a number greater than the number of grids "4" in step ST2 and executes the selection process (step ST3). In step ST3, as an example, the case where the number of grids is "16" is shown. For example, in step ST3, the processing unit 12 selects 15 data items. The processing unit 12 can, for example, reduce the number of grids by increasing the length of one side of each grid, or increase the number of grids by shortening the length of one side of each grid.

[0025] Next, the processing unit 12 determines whether the total number of data selected in the selection process of step ST3, "15", is the same as the specified number, "8". The processing unit 12 determines that the total number "15" is different from the specified number, "8". The processing unit 12 compares the total number "15" with the specified number, "8", and detects that the total number "15" is greater than the specified number, "8". Then, the processing unit 12 changes the number of grids to a number smaller than the number of grids "16" in step ST3 and executes the selection process (step ST4). At this time, the processing unit 12 changes the number of grids to a number larger than the number of grids "4" in step ST2. In step ST4, as an example, the case where the number of grids is "9" is shown. For example, in step ST4, the processing unit 12 selects 8 data.

[0026] The processing unit 12 determines whether the total number of data selected in the selection process of step ST4, "8", is the same as the specified number "8". The processing unit 12 determines that the total number "8" is the same as the specified number "8". Therefore, the processing unit 12 outputs information indicating the data selected in the selection process of step ST4 as the extraction result of data screening.

[0027] As described above, the processing unit 12 may generate multiple data representing points in N-dimensional space by projecting multiple original data representing points in M-dimensional space onto N-dimensional space through dimensionality reduction. In this case, the multiple N-dimensional data are associated one-to-one with the multiple M-dimensional original data. Therefore, the final extraction result from the above selection process (for example, the index of the extracted data) can be used as the result of data screening on the multiple M-dimensional original data.

[0028] In this way, the information processing device 10 obtains multiple data points, each representing a point in an N-dimensional space, and a specified number of data points to be extracted from the multiple data points. The N-dimensional space is divided into a first number of areas. For each of the first number of areas, a selection process is performed to select one data point from the data points present in that area. It is determined whether the total number of selected data points is equal to the specified number. If the total number is not equal to the specified number, the number of areas dividing the N-dimensional space is changed to a second number different from the first number, and the selection process is performed again.

[0029] This allows the information processing device 10 to extract data more efficiently. For example, since the information processing device 10 extracts one data item at a time from each non-overlapping area of ​​the N-dimensional space, it can ensure data diversity in the extraction results.

[0030] Here, as a comparative example, one could consider a method that calculates the distance between each pair of data, extracts one of the pairs with relatively close distances, and discards the other. For example, Euclidean distance can be used to calculate the distance between data. However, in the comparative example method, the distance is calculated for each and every pair of data. The computational complexity of this comparative example is O(n) for the total number of data points n. ２ Therefore, the larger the size of the data being screened, the greater the computational cost becomes.

[0031] In contrast, the information processing device 10 only needs to calculate which grid (area) each of the n data points belongs to. Therefore, the computational complexity of data screening by the information processing device 10 is O(n). As a result, the information processing device 10 can reduce the computational complexity and speed up data extraction compared to the comparative example method. Furthermore, even if the size of the data to be screened increases, the information processing device 10 can prevent the computational complexity from becoming excessive.

[0032] Furthermore, the methods in the above comparative examples often fail to extract the same number of data points as specified by the user. In contrast, the information processing device 10 can extract the same number of data points as specified by the user.

[0033] As mentioned above, the information processing device 10 can generate multiple data points in an N-dimensional space by projecting multiple original data points, each representing a point in an M-dimensional space, into an N-dimensional space through dimensionality reduction, and can perform selection processing on these multiple data points. As a result, the information processing device 10 can reduce computational costs and speed up data extraction compared to performing selection processing using multiple original data points in an M-dimensional space.

[0034] [Second Embodiment] Next, a second embodiment will be described. Figure 2 is a diagram showing an example of the hardware of the information processing device according to the second embodiment.

[0035] The information processing device 100 performs data screening on a dataset containing multiple data. The information processing device 100 may also be called a computer. The information processing device 100 has a processor 101, RAM 102, HDD 103, GPU 104, input interface 105, media reader 106, and communication interface 107. These units of the information processing device 100 are connected to a bus inside the information processing device 100. The processor 101 corresponds to the processing unit 12 of the first embodiment. The RAM 102 or HDD 103 corresponds to the storage unit 11 of the first embodiment.

[0036] The processor 101 is an arithmetic unit that executes program instructions. The processor 101 is, for example, a CPU. The processor 101 loads at least a portion of the program and data stored in the HDD 103 into the RAM 102 and executes the program. The processor 101 may include multiple processor cores. The information processing device 100 may have multiple processors. The processor that executes one of the multiple processes performed by the information processing device 100 may be different from the processor that executes a different process from the multiple processes. In addition, at least a portion of the processes described below may be executed in parallel using multiple processors or processor cores. A collection of multiple processors is sometimes called a "multiprocessor" or simply a "processor". A processor may also be called a "processor circuitry".

[0037] RAM 102 is a volatile semiconductor memory that temporarily stores programs executed by the processor 101 and data used by the processor 101 for calculations. The information processing device 100 may also be equipped with other types of memory, and may be equipped with multiple types of memory.

[0038] The HDD 103 is a non-volatile storage device that stores software programs such as the OS (Operating System), middleware, and application software, as well as data. The information processing device 100 may also be equipped with other types of storage devices such as flash memory or SSD (Solid State Drive), and may be equipped with multiple non-volatile storage devices.

[0039] The GPU 104 outputs an image to the display 111 connected to the information processing device 100, according to instructions from the processor 101. Any type of display can be used as the display 111, such as a CRT (Cathode Ray Tube) display, a liquid crystal display (LCD), a plasma display, or an organic electro-luminescence (OEL) display.

[0040] The input interface 105 acquires an input signal from an input device 112 connected to the information processing apparatus 100 and outputs it to the processor 101. As the input device 112, a pointing device such as a mouse, a touch panel, a touch pad, a track ball, a keyboard, a remote controller, a button switch, or the like can be used. Also, a plurality of types of input devices may be connected to the information processing apparatus 100.

[0041] The medium reader 106 is a reading device that reads a program or data recorded on a recording medium 113. As the recording medium 113, for example, a magnetic disk, an optical disk, a magneto-optical disk (MO: Magneto-Optical disk), a semiconductor memory, or the like can be used. The magnetic disk includes a flexible disk (FD: Flexible Disk) and an HDD. The optical disk includes a CD (Compact Disc) and a DVD (Digital Versatile Disc).

[0042] The medium reader 106, for example, copies a program or data read from the recording medium 113 to another recording medium such as the RAM 102 or the HDD 103. The read program is executed, for example, by the processor 101. Note that the recording medium 113 may be a portable recording medium and may be used for distribution of a program or data. Also, the recording medium 113 or the HDD 103 may be referred to as a computer-readable recording medium.

[0043] The communication interface 107 is connected to a network 114 and communicates with other information processing apparatuses via the network 114. The communication interface 107 may be a wired communication interface connected to a wired communication device such as a switch or a router, or may be a wireless communication interface connected to a wireless communication device such as a base station or an access point.

[0044] Figure 3 shows an example of data screening. The information processing device 100 improves the quality of training data by performing data screening on a dataset used, for example, as training data in machine learning. For example, the diversity of data in the training data affects the generalization performance of the machine learning model created based on the training data. Therefore, in order to improve the quality of the training data, the information processing device 100 uniformly screens the dataset in a way that ensures data diversity during data screening.

[0045] Figure 3 illustrates the original dataset 30 and dataset 31 after data screening of dataset 30. Each data point in dataset 30 represents a point in N-dimensional space, where N is an integer greater than or equal to 1. In Figure 3, points corresponding to each data point in datasets 30 and 31 are shown as solid circles. For example, dataset 30 contains a total of 45 data points. The information processing device 100 obtains dataset 31 by uniformly thinning the data in dataset 30. For example, dataset 31 contains a total of 15 data points. In the figure of dataset 31, data points from dataset 30 that are not included in dataset 31 are represented by dotted circles. The information processing device 100 provides a function to efficiently perform data screening.

[0046] Figure 4 shows an example of the functions of an information processing device. The information processing device 100 has a data storage unit 120, a dimensionality reduction processing unit 130, and a data extraction unit 140. The data storage unit 120 uses the storage area of ​​RAM 102 or HDD 103. The dimensionality reduction processing unit 130 and the data extraction unit 140 are realized by the execution of a program stored in RAM 102 by the processor 101.

[0047] The data storage unit 120 stores a plurality of original data indicating points in the M-dimensional space. M is an integer greater than N. The plurality of original data is a data set before dimensionality reduction that is input to the information processing apparatus 100 by a user or the like. Each of the plurality of original data has an index for identifying each original data. The index is a number such as 1, 2, 3, ….

[0048] Also, the data storage unit 120 stores a plurality of data indicating points in the N-dimensional space, which are generated by the dimensionality reduction processing unit 130 based on the plurality of original data. The plurality of data is a data set after dimensionality reduction. The data projected onto the N-dimensional space has the same index as the original data corresponding to the data.

[0049] The dimensionality reduction processing unit 130 performs dimensionality reduction processing on the plurality of original data in the M dimension to generate a plurality of data obtained by projecting the plurality of original data onto the N-dimensional space. For the dimensionality reduction processing, techniques such as principal component analysis and singular value decomposition are used, for example.

[0050] The data extraction unit 140 receives the number (designated number) of data to be extracted, which is input by the user, and extracts the received designated number of data from the plurality of N-dimensional data output by the dimensionality reduction processing unit 130.

[0051] Specifically, the data extraction unit 140 divides the N-dimensional space into a plurality of areas by dividing the N-dimensional space with a first number of N-dimensional grids. The data extraction unit 140 can divide the N-dimensional space into the same number of areas as the number of grids by arranging each grid adjacent to each other in the N-dimensional space without gaps. For each of the plurality of areas, the data extraction unit 140 performs a selection process of selecting one data from the data existing in the area among the plurality of data.

[0052] The data extraction unit 140 determines whether the total number of data selected in the current selection process is the same as a specified number. If the total number is the same as the specified number, the data extraction unit 140 outputs information indicating the data selected in the current selection process as the extraction result of data screening. If the total number is not the same as the specified number, the data extraction unit 140 changes the number of grids dividing the N-dimensional space to a second number of grids different from the first number and executes the selection process.

[0053] Next, the procedure for data screening by the information processing device 100 will be explained. In the following explanation, a point represented by a piece of data in N-dimensional space will be referred to as a data point. Selecting a data point and selecting data are essentially the same thing.

[0054] First, let's consider the case where N=2. The two-dimensional space is represented by orthogonal X and Y axes. Data points are represented by X and Y coordinates. The grid dividing the two-dimensional space is a square. Figure 5 is a flowchart of an example of data screening.

[0055] (S10) The data extraction unit 140 receives information on the specified number of data to be extracted by data screening and information specifying the dataset to be screened. In this example, the number of dimensions M of the dataset specified in step S10 is 3 or more. The data extraction unit 140 stores the information on the specified number of data in the data storage unit 120 and notifies the dimensionality reduction processing unit 130 of the information on the specified dataset. The dimensionality reduction processing unit 130 retrieves the specified dataset from the data storage unit 120 and projects the dataset onto a two-dimensional space. Specifically, the dimensionality reduction processing unit 130 generates multiple data representing points in a two-dimensional space by performing dimensionality reduction processing on multiple original data representing points in an M-dimensional space.

[0056] (S11) The data extraction unit 140 sets the variable D_low to 0 and the variable D_high to the maximum values ​​of the X-axis and Y-axis lengths. The length of the X-axis is, for example, the difference between the maximum and minimum values ​​of the X coordinate at each data point. The length of the Y-axis is, for example, the difference between the maximum and minimum values ​​of the Y coordinate at each data point.

[0057] (S12) The data extraction unit 140 divides the two-dimensional space into a grid, with the length S of one side of the grid being (D_low + D_high) / 2. (S13) The data extraction unit 140 selects the data point with the smallest index in each grid.

[0058] (S14) The data extraction unit 140 determines whether the number of data points selected in step S13 is the same as the number specified in step S10 (specified number). If the number of data points selected in step S13 is different from the specified number, the process proceeds to step S15. If the number of data points selected in step S13 is the same as the specified number, the data extraction unit 140 outputs the index corresponding to each data point selected in step S13 as the data screening result, and terminates the data screening process.

[0059] (S15) The data extraction unit 140 determines whether the number of data points selected in step S13 is less than the number specified in step S10 (specified number). If the number of data points selected in step S13 is less than the specified number, the process proceeds to step S16. If the number of data points selected in step S13 is greater than the specified number, the process proceeds to step S17.

[0060] (S16) The data extraction unit 140 updates D_high = S. Then the process proceeds to step S12. (S17) The data extraction unit 140 updates D_low = S. Then the process proceeds to step S12.

[0061] In step S13, the data extraction unit 140 may select the data point with the highest index in each grid, or it may select the data point with the median index in the relevant grid. Alternatively, the data extraction unit 140 may select the data closest to the center point of the relevant grid, or it may select the data closest to the centroid of each data point included in the relevant grid.

[0062] Furthermore, the data extraction unit 140 can efficiently determine the grid size by automatically calculating it using binary search, as shown in steps S15 to S17. Alternatively, for example, the data extraction unit 140 may obtain data corresponding to the index finally output as a result of data screening from the original M-dimensional dataset and use each of the obtained data as training data in machine learning. This allows the information processing device 100 to improve the quality of the training data. For example, the information processing device 100 can improve the generalization performance of the machine learning model created by machine learning based on the training data.

[0063] Next, a specific example of data screening by the information processing device 100 will be explained. Figure 6 shows an example of data screening. Multiple data points are plotted in the two-dimensional space 40. The two-dimensional space 40 is a rectangular region that encloses the multiple data points. The length of the X axis of the two-dimensional space 40 is A. The length of the Y axis of the two-dimensional space 40 is B. The maximum value of the length of the X axis A and the length of the Y axis B is assumed to be B. Also, the number of data specified by the user is assumed to be "10".

[0064] When step S12 is performed for the first time, i.e., in iteration "0", D_low = 0 and D_high = B. Therefore, the data extraction unit 140 sets S = (0 + B) / 2 = B / 2. The data extraction unit 140 then divides the two-dimensional space 40 into four areas using four grids, each with a side length S of S = B / 2. The data extraction unit 140 selects the data point with the minimum index from each of these four areas. The data points selected from each area are called "representative points". In the figure, the representative points are shown as black circles plotted in the corresponding area within the grid.

[0065] In the example in Figure 6, the number of representative points in iteration "0" is 4. The number of representative points "4" is less than the specified number "10". Therefore, the data extraction unit 140 updates D_high = S = B / 2.

[0066] Next, when step S12 is executed, i.e., in iteration "1", D_low = 0 and D_high = B / 2. Therefore, the data extraction unit 140 sets S = (0 + B / 2) / 2 = B / 4. The data extraction unit 140 then divides the two-dimensional space 40 into 16 areas using 16 grids, each with a side length S of S = B / 4. The data extraction unit 140 selects the data point with the minimum index from each of these 16 areas.

[0067] In the example in Figure 6, the number of representative points in iteration "1" is 14. The number of representative points "14" is greater than the specified number "10". Therefore, the data extraction unit 140 updates D_low = S = B / 4.

[0068] Next, when step S12 is executed, i.e., in iteration "2", D_low = B / 4 and D_high = B / 2. Therefore, the data extraction unit 140 sets S = (B / 4 + B / 2) / 2 = (3 / 8)B. The data extraction unit 140 then divides the two-dimensional space 40 into nine areas using nine grids, each with a side length S of S = (3 / 8)B. The data extraction unit 140 selects the data point with the minimum index from each of these nine areas.

[0069] In the example in Figure 6, the number of representative points in iteration "2" is 8. The number of representative points "8" is less than the specified number "10". Therefore, the data extraction unit 140 updates D_high = S = (3 / 8)B. Subsequently, the data extraction unit 140 repeats the above procedure, and when the number of representative points matches the specified number "10", it outputs the index of each representative point that was finally selected and terminates the data screening.

[0070] By the way, the following comparative example method can be considered as a way to perform screening while ensuring data diversity. The device that executes the comparative example will be called an "information processing device." Figure 7 is a flowchart of the comparative example.

[0071] (S100) The information processing device receives information on the specified number of data to be extracted by data screening and information specifying the dataset to be screened. The information processing device projects the dataset onto a two-dimensional space.

[0072] (S101) The information processing device calculates the Euclidean distance between each data point projected onto the two-dimensional space and other data points. That is, the information processing device calculates the Euclidean distance for each pair of data points.

[0073] (S102) The information processing device calculates the minimum value of the Euclidean distance calculated in step S101 and sets this minimum value as D_threhold. (S103) For pairs of data with a Euclidean distance of (D_threhold) to (D_threhold+α), the information processing device deletes one of the data points. The threshold α is a positive real number. The value of α is predetermined. In one example, α = 0.01.

[0074] (S104) The information processing device determines whether the number of selected data points, that is, the number of data points that have not been deleted through the repetition of step S103, is less than or equal to the specified number of data points (specified number). If the number of selected data points is less than or equal to the specified number, the information processing device outputs the index of each data point that has not been deleted and terminates the data screening. If the number of selected data points is greater than the specified number of data points, the process proceeds to step S105.

[0075] (S105) The information processing device updates D_threhold to += α. That is, the information processing device adds α to the current value of D_threhold and sets that value as the new value of D_threhold. Then the process proceeds to step S103.

[0076] Reference 1 is relevant to the comparative example in Figure 7. Reference 1: K. Shimizu, et al., “Phase stability of Au-Li binary systems studied using neural network”, [online], 2021, Phys. Rev. B 103, 094112 (2021), [Retrieved October 9, 2024], Internet <URL: https: / / journals.aps.org / prb / pdf / 10.1103 / PhysRevB.103.094112> In the procedure of the above comparative example, the Euclidean distance is calculated for all pairs of data. Therefore, if the size of the dataset to be screened, i.e., the number of data points in the dataset, is large, the computational cost of data extraction becomes excessive.

[0077] Next, we will illustrate the data screening results obtained by the information processing device 100 and the data screening results obtained by the comparative example method. Figure 8 shows an example of the data screening results.

[0078] Table 50 shows the experimental results of data screening using the information processing device 100 and the comparative example methods, when the number of data points n in the dataset is set to n = 10,000 and n = 100,000. The number of data points after screening (specified number) is set to 1,000. The processor used in the experiment was an Intel® Xeon® CPU E5-2698 v4, with one core and one thread.

[0079] In both cases, n=10,000 and n=100,000, each data point (population) in the dataset is generated using NumPy 1.26.4 so that it follows a normal distribution with mean 0 and standard deviation 1. In Table 50, the rows for the normal population show the standard deviation values ​​for each data point in both n=10,000 and n=100,000 cases, while other items (number of extracted data points and execution time) are marked with a hyphen ("-") to indicate no value.

[0080] For comparison, Table 50 also includes the results of random sampling. Random sampling is a method of randomly selecting data from n items. In Table 50, the number of extracted data, standard deviation, and execution time are listed for each row of the random sampling method, the comparative example method, and the information processing device 100. The number of extracted data is the number of data ultimately extracted from the normal population by data screening. The standard deviation is the standard deviation of each data extracted by data screening. The execution time is the time required to perform the data screening. The unit of execution time is seconds (s).

[0081] The data screening results for each method when n = 10,000 are as follows: For random sampling, the number of extracted data points was "1,000", the standard deviation was "0.9974", and the execution time was "0.0011" seconds.

[0082] In the comparative example method illustrated in Figure 7, the number of extracted data points was "911", the standard deviation was "1.5368", and the execution time was "3.6239" seconds. In the information processing device 100 method illustrated in Figure 5, the number of extracted data points was "1000", the standard deviation was "1.5265", and the execution time was "0.4794" seconds.

[0083] The data screening results for each method when n = 100,000 are as follows: For random sampling, the number of extracted data points was "1,000", the standard deviation was "1.0051", and the execution time was "0.0127" seconds.

[0084] In the comparative example method illustrated in Figure 7, the number of extracted data points was "927", the standard deviation was "1.9069", and the execution time was "419.7760" seconds. In the information processing device 100 method illustrated in Figure 5, the number of extracted data points was "1000", the standard deviation was "1.9101", and the execution time was "4.4169" seconds.

[0085] According to Table 50, the method of the information processing device 100 yields a larger standard deviation for each data point obtained than random sampling. In other words, the method of the information processing device 100 allows for the extraction of more diverse data than random sampling.

[0086] Furthermore, according to Table 50, the method of the information processing device 100 has a shorter execution time than the method of the comparative example. In other words, the method of the information processing device 100 can shorten the execution time of data sampling compared to the method of the comparative example. Also, in the comparative example method, the number of extracted data is less than the specified number. On the other hand, the method of the information processing device 100 can appropriately extract the number of data specified by the user.

[0087] Thus, according to the data screening method of the information processing device 100, a number of data specified by the user can be obtained in a short time while maintaining diversity. Figure 9 shows an example of the data screening results.

[0088] Graph 60 is a scatter plot showing the normal population used in the experiment for n = 10,000 in Figure 8. Here, a scatter plot plots each data point in a two-dimensional space. Graph 61 shows the data set extracted as a result of random sampling of the normal population in Graph 60. Graph 62 shows the data set extracted as a result of data screening of the normal population in Graph 60 using the comparative example method. Graph 63 shows the data set extracted as a result of data screening of the normal population in Graph 60 using the information processing device 100 method.

[0089] Graphs 61, 62, and 63 show that the information processing device 100 can extract diverse data from the original dataset. Figure 10 shows an example of the data screening results.

[0090] Graph 70 shows the normal population used in the experiment for n = 100,000 in Figure 8. Graph 71 shows the data set extracted as a result of random sampling from the normal population in Graph 70. Graph 72 shows the data set extracted as a result of data screening using the comparative example method for the normal population in Graph 70. Graph 73 shows the data set extracted as a result of data screening using the information processing device 100 method for the normal population in Graph 70.

[0091] Graphs 71, 72, and 73 show that the information processing device 100 can extract diverse data from the original dataset. Figure 11 illustrates the computational complexity of data screening.

[0092] The data point cloud 80 represents n = 6 data points. The comparative example method in Figure 7 calculates the Euclidean distance for each combination of data points. The computational complexity of the comparative example method is O(n ２ ) This is the result. In the comparative example method, for example, 36 elements in a 6x6 Euclidean distance matrix 81 are calculated for the data point cloud 80.

[0093] On the other hand, the method of the information processing device 100 calculates which grid each point is contained in. The computational complexity of the method of the information processing device 100 is O(n). For example, the data extraction unit 140 may generate grid index information 82 for each of the six data points, indicating the index of the grid in which the data point is contained. In this case, the data extraction unit 140 scans the grid index information 82 starting from the smallest data point index, and when the index of the first grid is found, it determines that the index of the data point at that time as the index of the representative point in that grid. The grid index information 82 may be stored in the data storage unit 120.

[0094] Thus, the information processing device 100 can reduce the computational load compared to the comparative example method and speed up the data screening process. Furthermore, the amount of information in the grid index information 82 is smaller than the amount of information in the Euclidean distance matrix 81. For this reason, the information processing device 100 can reduce the amount of memory used for calculations, such as RAM 102.

[0095] Another comparative example of data screening is a technique called Voxelization Down-sampling, which is used, for example, to reduce the data volume of 3D point cloud data such as 3D models. Reference 2 is a useful reference for Voxelization Down-sampling.

[0096] Document 2: Dawei Li, 3 other authors, "PSegNet: Simultaneous Semantic and Instance Segmentation for Point Clouds of Plants", [online], 2022, Plant Phenomics, [searched on October 9, 2024], Internet <URL: https: / / spj.science.org / doi / 10.34133 / 2022 / <9787643> Figure 12 is a diagram showing other comparative examples.

[0097] In Voxelization Down-sampling, the data point cloud is divided by a three-dimensional grid (voxel), and the centroid of each data point within the voxel is calculated. Then, each data point in the voxel is replaced by the centroid point. That is, in Voxelization Down-sampling, new data points (centroid points) are generated, and the information of each original data point is deleted. Voxelization Down-sampling does not select existing data points.

[0098] Therefore, Voxelization Down-sampling cannot be applied to cases where "you want to select data from a data point cloud". For example, Voxelization Down-sampling cannot be applied for "improving the quality of training data in machine learning".

[0099] Also, in Voxelization Down-sampling, the voxel size is predetermined. The number of data points after screening is determined depending on the voxel size. In Voxelization Down-sampling, the voxel size is set manually, and the number of data points after screening cannot be determined in advance.

[0100] Figure 13 shows an example of data selection by an information processing device according to the second embodiment. The information processing device 100 selects data points from areas divided by each grid, for example, based on an index. In one example, the information processing device 100 selects the data point with the smallest index among the data points belonging to that area. Therefore, the information processing device 100 can select existing data points. In addition, the information processing device 100 can specify in advance the number of data points to be extracted, and can extract the same number of data points as the specified number while maintaining the diversity of data points.

[0101] The information processing device 100 may also select the data point with the highest index among the data points belonging to the area, or the data point with the median index among the data points belonging to the area. The information processing device 100 may also select the data point closest to the centroid of the data points belonging to the area, or the data point closest to the center of the area.

[0102] [Modifications] Next, modifications of the second embodiment will be described. Figure 5 illustrates processing of data points in a two-dimensional space, but the information processing device 100 can perform similar processing on data points in one-dimensional or three-dimensional or more-dimensional spaces.

[0103] For example, when projecting data points into three-dimensional space, the following procedure is followed: Three-dimensional space is represented by orthogonal X, Y, and Z axes. Data points are represented by X, Y, and Z coordinates. The grid that divides the three-dimensional space is a cube.

[0104] Figure 14 is a flowchart showing the first modified example. (S20) The data extraction unit 140 receives information on the specified number of data to be extracted by data screening and information specifying the dataset to be screened. In this example, the number of dimensions M of the dataset specified in step S20 is 4 or more. The data extraction unit 140 stores the information on the specified number of data in the data storage unit 120 and notifies the dimensionality reduction processing unit 130 of the information on the specified dataset. The dimensionality reduction processing unit 130 retrieves the specified dataset from the data storage unit 120 and projects the dataset onto a three-dimensional space. Specifically, the dimensionality reduction processing unit 130 generates multiple data representing points in a three-dimensional space by performing dimensionality reduction processing on multiple original data representing points in an M-dimensional space.

[0105] (S21) The data extraction unit 140 sets the variable D_low to 0 and the variable D_high to the maximum values ​​of the lengths of the X, Y, and Z axes. The length of the X axis is, for example, the difference between the maximum and minimum values ​​of the X coordinate at each data point. The length of the Y axis is, for example, the difference between the maximum and minimum values ​​of the Y coordinate at each data point. The length of the Z axis is, for example, the difference between the maximum and minimum values ​​of the Z coordinate at each data point.

[0106] (S22) The data extraction unit 140 divides the three-dimensional space into a grid, with the length S of one side of the grid being (D_low + D_high) / 2. (S23) The data extraction unit 140 selects the data point with the smallest index in each grid.

[0107] (S24) The data extraction unit 140 determines whether the number of data points selected in step S23 is the same as the number specified in step S20 (specified number). If the number of data points selected in step S23 is different from the specified number, the process proceeds to step S25. If the number of data points selected in step S23 is the same as the specified number, the data extraction unit 140 outputs the index corresponding to each data point selected in step S23 as the data screening result and terminates the data screening process.

[0108] (S25) The data extraction unit 140 determines whether the number of data points selected in step S23 is less than the number specified in step S20 (specified number). If the number of data points selected in step S23 is less than the specified number, the process proceeds to step S26. If the number of data points selected in step S23 is greater than the specified number, the process proceeds to step S27.

[0109] (S26) The data extraction unit 140 updates D_high = S. Then the process proceeds to step S22. (S27) The data extraction unit 140 updates D_low = S. Then the process proceeds to step S22.

[0110] For example, when projecting data points into an N-dimensional space, the following procedure is followed: If N ≥ 2, the N-dimensional space is represented by N mutually orthogonal axes. If N = 1, the N-dimensional space is represented by one axis. Data points are represented by N coordinates corresponding to the N axes. The grid that divides the N-dimensional space is an N-dimensional hypercube.

[0111] Figure 15 is a flowchart showing a second modified example. (S30) The data extraction unit 140 receives information on the specified number of data to be extracted by data screening and information specifying the dataset to be screened. In this example, the number of dimensions M of the dataset specified in step S20 is greater than N. The data extraction unit 140 stores the information on the specified number of data in the data storage unit 120 and notifies the dimensionality reduction processing unit 130 of the information on the specified dataset. The dimensionality reduction processing unit 130 retrieves the specified dataset from the data storage unit 120 and projects the dataset onto an N-dimensional space. Specifically, the dimensionality reduction processing unit 130 generates multiple data representing points in an N-dimensional space by performing dimensionality reduction processing on multiple original data representing points in an M-dimensional space.

[0112] (S31) The data extraction unit 140 sets the variable D_low to 0 and the variable D_high to the value of the longest side among the sides corresponding to the N-dimensional axes. The length of the side corresponding to a certain axis is, for example, the difference between the maximum and minimum values ​​of the coordinates of that axis at each data point.

[0113] (S32) The data extraction unit 140 divides the N-dimensional space into a grid, with the length S of one side of the grid being (D_low + D_high) / 2. (S33) The data extraction unit 140 selects the data point with the smallest index in each grid.

[0114] (S34) The data extraction unit 140 determines whether the number of data points selected in step S33 is the same as the number specified in step S30 (specified number). If the number of data points selected in step S33 is different from the specified number, the process proceeds to step S35. If the number of data points selected in step S33 is the same as the specified number, the data extraction unit 140 outputs the index corresponding to each data point selected in step S33 as the data screening result and terminates the data screening process.

[0115] (S35) The data extraction unit 140 determines whether the number of data points selected in step S33 is less than the number specified in step S30 (specified number). If the number of data points selected in step S33 is less than the specified number, the process proceeds to step S36. If the number of data points selected in step S33 is greater than the specified number, the process proceeds to step S37.

[0116] (S36) The data extraction unit 140 updates D_high = S. Then the process proceeds to step S32. (S37) The data extraction unit 140 updates D_low = S. Then the process proceeds to step S32.

[0117] In steps S23 and S33, the data extraction unit 140 may select the data point with the highest index in each grid, or it may select the data point with the median index in the relevant grid. Alternatively, the data extraction unit 140 may select the data closest to the center point of the relevant grid, or it may select the data closest to the centroid of each data point included in the relevant grid.

[0118] For example, the data extraction unit 140 may obtain data corresponding to the index ultimately output as a result of data screening from the original M-dimensional dataset, and use each of the obtained data as training data in machine learning.

[0119] According to the information processing device 100 of the second embodiment, data extraction can be made more efficient. For example, the information processing device 100 can extract the same number of data as a specified number (specified number) while ensuring diversity. In addition, the information processing device 100 can speed up the extraction of said data.

[0120] The information processing device 100 performs, for example, the following processes: The data extraction unit 140 receives a specified number of data to be extracted from a plurality of data, each representing a point in an N-dimensional space. The data extraction unit 140 divides the N-dimensional space into a first number of areas. For each of the first number of areas, the data extraction unit 140 performs a selection process to select one data from among the plurality of data present in that area. The data extraction unit 140 determines whether the total number of selected data is the same as the specified number. If the total number is not the same as the specified number, the data extraction unit 140 changes the number of areas dividing the N-dimensional space to a second number different from the first number and performs the selection process.

[0121] This allows the information processing device 100 to extract data more efficiently. For example, the information processing device 100 can extract data at high speed. Also, for example, the information processing device 100 can extract the same number of data as a specified number while maintaining the diversity of the original data set. Furthermore, the information processing device 100 can reduce the amount of memory used, such as RAM 102, for calculations.

[0122] For example, the area of ​​the first number may be a partitioned area of ​​N-dimensional space divided by an N-dimensional grid of the first number. Each of the multiple data points may have an index that identifies it. In the selection process, the data extraction unit 140 may select one data point from among the data points present in the area based on the index of each data point present in that area.

[0123] This allows the information processing device 100 to extract data more efficiently. For example, the information processing device 100 can extract data at high speed. As a method for selecting data based on an index, for example, one could select the data with the smallest or largest index, or the data with the median index, from among the data present in the area. However, it is preferable to minimize the amount of computation required for data selection. For this reason, the method of selecting the data with the smallest or largest index from among the data present in the area is more preferable. The data extraction unit 140 can easily obtain the data with the smallest or largest index for each area by scanning the aforementioned grid index information 82 in ascending or descending order of the data index.

[0124] For example, when changing the number of grid cells, the data extraction unit 140 compares the total number of selected data with the specified number. If the total number is less than the specified number, the data extraction unit 140 sets the second number to a number greater than the first number. If the total number is greater than the specified number, the data extraction unit 140 sets the second number to a number less than the first number.

[0125] This allows the information processing device 100 to efficiently bring the number of data selected in the selection process closer to a specified number. As mentioned above, the data extraction unit 140 can increase the number of grids by shortening the length of one side of the grid. Conversely, the data extraction unit 140 can decrease the number of grids by lengthening the length of one side of the grid.

[0126] Furthermore, as shown in Figure 6, when the data extraction unit 140 decreases the number of grids after increasing them, for example, it determines the number of grids so that it is greater than the number of grids before the previous increase. When the data extraction unit 140 increases the number of grids after decreasing them, it determines the number of grids so that it is smaller than the number of grids before the previous decrease. More specifically, in iteration "2" in Figure 6, the number of grids is determined to be "9", which is greater than the number of grids "4" before the previous increase (in iteration "0"). Also, if the number of grids is increased in the iteration following iteration "2" in Figure 6, the number of grids in that next iteration will be determined to be smaller than the number of grids "14" before the previous decrease (in iteration "1"). Alternatively, if the number of grids is reduced in the iteration following iteration "2" in Figure 6, the number of grids in that next iteration will be determined to be greater than the number of grids "4" before the previous increase (in iteration "0").

[0127] This allows the information processing device 100 to efficiently determine the number of grid cells. For example, the information processing device 100 can quickly approach a specified number of selected data cells, thereby speeding up data extraction.

[0128] The data extraction unit 140 outputs information indicating the selected data as an extraction result if the total number of data selected in the selection process is the same as the specified number. This allows the information processing device 100 to easily identify each data after data screening of multiple data, and makes each data after data screening available for use as training data in machine learning, etc.

[0129] The dimensionality reduction processing unit 130 may generate multiple data by projecting each of the multiple original data, each representing a point in M-dimensional space, onto N-dimensional space. This allows the information processing device 100 to reduce the computational cost of the data screening process and speed up the processing compared to using the multiple original data in M-dimensional space directly. Furthermore, the information processing device 100 can reduce the amount of data used for calculations by reducing dimensionality, thereby reducing the memory usage of RAM 102 and other memory. For example, by making the number of dimensions relatively small, such as N=1 or N=2, further speed improvements and memory savings can be achieved. For projection from M-dimensional space to N-dimensional space, methods such as principal component analysis and singular value decomposition can be used.

[0130] Furthermore, the dimensionality reduction processing unit 130 and the data extraction unit 140 in the information processing device 100 may be executed by one or more processors. For example, the processor executing one of these processes may be different from the processor executing another. Furthermore, at least a portion of the processes of the information processing device 100 may be executed in parallel by multiple processor cores or multiple processors. For example, the selection process may be executed in parallel by multiple processor cores or multiple processors.

[0131] Furthermore, the information processing in the first embodiment can be achieved by having the processing unit 12 execute a program. Similarly, the information processing in the second embodiment can be achieved by having the processor 101 execute a program. The program can be recorded on a computer-readable recording medium 113.

[0132] For example, a program can be distributed by distributing a recording medium 113 on which the program is stored. Alternatively, the program may be stored on another computer and distributed via a network. A computer may, for example, store (install) a program stored on the recording medium 113 or a program received from another computer into a storage device such as RAM 102 or HDD 103, and then read and execute the program from that storage device.

[0133] The above merely illustrates the principle of the present invention. Furthermore, numerous modifications and changes are possible for those skilled in the art, and the present invention is not limited to the exact configurations and applications shown and described above. All corresponding modifications and equivalents are considered to be within the scope of the present invention as defined by the appended claims and their equivalents.

[0134] 10 Information processing device 11 Storage unit 12 Processing unit 20 N-dimensional space 21, 22, 23, 24 Grid 21a, 22a, 23a, 24a Area

Claims

1. A data screening program that causes a computer to receive a specified number of data to be extracted from a plurality of data, each representing a point in an N-dimensional space (where N is an integer greater than or equal to 1); divide the N-dimensional space into a first number of areas; for each of the first number of areas, perform a selection process to select one data from the plurality of data present in that area; determine whether the total number of selected data is the same as the specified number; and if the total is not the same as the specified number, change the number of areas dividing the N-dimensional space to a second number different from the first number and execute the selection process.

2. The data screening program according to claim 1, wherein the area of ​​the first number is divided by dividing the N-dimensional space into an N-dimensional grid of the first number, each of the plurality of data has an index that identifies the data, and the selection process causes the computer to perform a process of selecting one data from among the data present in the area based on the index of each data present in the area.

3. The data screening program according to claim 2, wherein the selection process causes the computer to perform a process of selecting the data with the smallest or largest index from among the data present in the area.

4. A data screening program according to claim 1, which causes the computer to perform the following processing when changing the number of areas: comparing the total number with the specified number; if the total number is less than the specified number, setting the second number to a number greater than the first number; and if the total number is greater than the specified number, setting the second number to a number smaller than the first number.

5. The data screening program according to claim 1, which causes the computer to perform a process that outputs information indicating the selected data as an extraction result when the total number is the same as the specified number.

6. A data screening program according to claim 1, which causes the computer to perform a process of generating a plurality of data by projecting each of the plurality of original data, each representing a point in an M-dimensional space (where M is an integer greater than N), onto the N-dimensional space.

7. A data screening method comprising: a computer receiving a specified number of data to be extracted from a plurality of data, each representing a point in an N-dimensional space (where N is an integer greater than or equal to 1); dividing the N-dimensional space into a first number of areas; performing a selection process for each of the first number of areas to select one data from the plurality of data present in that area; determining whether the total number of selected data is the same as the specified number; and, if the total is not the same as the specified number, changing the number of areas dividing the N-dimensional space to a second number different from the first number and executing the selection process.

8. An information processing device comprising: a storage unit that stores a plurality of data, each representing a point in an N-dimensional space (where N is an integer greater than or equal to 1); and a processing unit that receives a specified number of data to be extracted from the plurality of data, divides the N-dimensional space into a first number of areas, performs a selection process for each of the first number of areas to select one data from the plurality of data present in that area, determines whether the total number of selected data is the same as the specified number, and if the total number is not the same as the specified number, changes the number of areas dividing the N-dimensional space to a second number different from the first number and executes the selection process.

Images