Server-Side Data Abstraction Layer for Distributed Column Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data manipulation strategies face challenges in efficiently handling data from disparate, non-uniform sources and lack scalability, particularly on client devices with limited processing and storage capabilities.
Innovation Solution
The implementation of a scalable frame data structure, SFrame, which is an immutable, column-store database with full query capabilities, allowing for efficient sub-selection of columns and distributed storage and processing across multiple servers, enabling efficient data manipulation and graph construction from diverse data sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored and processed on client devices, then processing and storage capabilities are sufficient, but client devices lack the processing and storage capabilities of larger server computing devices
Solution Approach 1:
A server-side data abstraction layer is introduced as an intermediary between client devices and remote servers. This layer provides familiar data structures (DataFrames, Series) and manipulation operations to clients, while handling the complexity of remote data access, distributed processing, and data integration automatically. Users can manipulate remote data as if it were local without needing to understand the underlying distributed system complexity.
2Adaptability or versatility
If data from multiple disparate sources is combined, then data integration is achieved, but it is difficult to manipulate data obtained from two or more disparate, non-uniform data sources in an efficient way
Solution Approach 1:
The system transforms heterogeneous data from disparate sources (CSV files, SQL databases, flat files) into a unified homogeneous structure using standardized DataFrames and Series objects. All data sources are converted to the same internal representation with consistent data types, indexing, and manipulation operations, enabling efficient processing regardless of the original source format.
Solution Approach 2:
The data abstraction layer provides universal DataFrames and Series structures that can represent and manipulate data from any source type through a single unified interface. The same manipulation operations (filtering, sorting, joining, aggregation) work uniformly across data from CSV files, databases, and other sources without requiring source-specific code.
3Productivity
If data is stored in traditional row-based databases, then complete rows are stored together, but efficient sub-selection of columns is not possible without loading remaining columns
Solution Approach 1:
The system segments data storage into columnar format where each column is stored separately in one or more files, rather than storing complete rows together. This segmentation enables the server to load and process only the specific columns requested by the client, avoiding the need to transfer entire rows even when only a subset of columns is needed, thereby reducing data loading volume and improving efficiency.
Data Source
AI summary
A method receives a first request from a client object at a device. The first request specifies a data source. In response to the first request, the method uploads data from the data source, stores the data as a plurality of first columns, and instantiates a first server object that provides access to the first columns. The method later receives a second request from the client object. The second request specifies a transformation of the data. In response to the second request, the method stores one or more additional columns and instantiates a second server object that provides access to the additional columns and one or more of the first columns. Each of the additional columns is constructed from the first columns according to the requested transformation, and each of the additional columns includes a plurality of data values all having the same data type.