Constraint-Based Cluster Selection for Large Data Point Sets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing clustering algorithms for data points, such as HDBSCAN, are computationally expensive and have high memory requirements, making them impractical for large datasets, while methods like the elbow method are subjective and lack clarity in parameter selection.
Innovation Solution
A system where candidate clusters and their properties are pre-computed at a computer server, with a client device selecting and visualizing clusters based on user-defined constraints, reducing computational and memory load on the client device.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If HDBSCAN clustering algorithm is used to achieve accurate clustering results, then clustering quality is improved, but computational cost and memory requirements increase significantly
Solution Approach 1:
The system performs preliminary clustering computations on the server to generate candidate clusters before the client device receives the data. This pre-computation approach allows the client to work with pre-processed clustering results rather than performing computationally expensive clustering operations locally, thereby reducing the computational burden on client devices while maintaining clustering quality.
Solution Approach 2:
The clustering process is divided into two segments: (1) server-side candidate cluster generation using HDBSCAN, and (2) client-side cluster selection from pre-generated candidates. This segmentation allows the computationally intensive portion to be performed once on the server, while the client performs lighter selection operations, resolving the contradiction between clustering quality and computational cost.
2Measurement precision
If HDBSCAN clustering algorithm is used to achieve accurate clustering results, then clustering quality is improved, but memory requirements increase significantly
Solution Approach 1:
The system extracts and transmits only the essential clustering results (candidate clusters with their properties) from the server to the client, rather than transmitting the entire dataset or intermediate computational structures. This extraction approach allows the client to achieve accurate clustering visualization with minimal memory requirements, as it only stores the final cluster assignments rather than the complete data or complex data structures required by HDBSCAN.
3Ease of operation
If clustering computations are performed at the client device to enable local processing, then processing autonomy is improved, but computational load and device requirements increase
Solution Approach 1:
The server acts as an intermediary that performs the computationally intensive clustering operations and provides processed results to the client. This intermediary approach allows the client to maintain processing autonomy for cluster selection and visualization while offloading the heavy computational load to the server, thereby reducing device complexity requirements without sacrificing operational independence.
4Loss of information
If the entire dataset is transmitted to the client device to enable complete data access, then data availability is improved, but transmission time and client memory increase
Solution Approach 1:
The system extracts and transmits only the essential information needed for clustering visualization - namely, the candidate cluster assignments and their properties - rather than transmitting the entire raw dataset. This extraction approach maintains data availability for clustering purposes while significantly reducing transmission time and client memory requirements, as only the processed cluster information is transferred.
Data Source
AI summary
A computer system and method for clustering data points comprises a computer server configured to: store a dataset compris-ing a plurality of data points; determine a plurality of candidate clusters for the plurality of data points, where each candidate cluster corresponds to a respec-tive set of the plurality of data points; determine, for each candidate cluster, one or more properties of that candidate cluster; and transmit, to the client device, a sample of the plurality of data points, information indicative of the plurality of candidate clusters, and information indicative of the one or more properties de-termined for each candidate cluster; and a client device configured to: receive the sample of data points, information indicative of the plurality of candidate clusters, and information indicative of the one or more properties determined for each candidate cluster; and determine a cluster constraint, wherein the cluster constraint relates to at least one of the one or more properties of each candidate cluster; select one or more candidate clusters from the plurality of candidate clusters, wherein the at least one property of each selected candidate cluster satisfies the determined cluster constraint.