Graph Data Binary Encoding to Reduce Shuffle and IO Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed graph computing engines face performance limitations due to high data shuffle and IO overheads as graph data scales increase.
Innovation Solution
Implement a binary coding scheme for graph data, converting attribute values into binary forms and storing them in a byte group sequence format that matches the number of attribute byte groups, reducing serialization and deserialization overheads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If graph data is stored in object form with schemas, then data structure and flexibility are maintained, but data shuffle overhead and IO overhead increase significantly
Solution Approach 1:
The patent changes the parameter of data representation from object form to binary form. By converting graph data attributes from structured objects to binary sequences, the system maintains the ability to represent diverse data types while dramatically reducing the overhead associated with data shuffle and IO operations in distributed computing environments.
Solution Approach 2:
The patent replaces the mechanical system of object-based data storage and serialization with a binary encoding system. This substitution eliminates the need for complex object serialization/deserialization processes during data shuffle operations, thereby reducing computational overhead while preserving data structure information through the binary format.
2Adaptability or versatility
If graph data is stored in object form with schemas, then data structure and flexibility are maintained, but IO overhead increases significantly
Solution Approach 1:
The patent changes the parameter of data representation from object form to binary form. By converting graph data attributes from structured objects to binary sequences, the system maintains the ability to represent diverse data types while dramatically reducing the overhead associated with data shuffle and IO operations in distributed computing environments.
Solution Approach 2:
The patent replaces the mechanical system of object-based data storage and serialization with a binary encoding system. This substitution eliminates the need for complex object serialization/deserialization processes during data shuffle operations, thereby reducing computational overhead while preserving data structure information through the binary format.
3Ease of operation
If attribute values are stored in object form, then data readability and maintainability are improved, but data processing efficiency decreases
Solution Approach 1:
The patent changes the parameter of data representation from object form to binary form. By converting graph data attributes from structured objects to binary sequences, the system maintains the ability to represent diverse data types while dramatically reducing the overhead associated with data shuffle and IO operations in distributed computing environments.
Data Source
AI summary
Embodiments of this specification provide a graph data processing method and apparatus, and a graph database-based data processing method and apparatus. In the graph data processing method, graph data represented in a data object form is obtained, where the graph data includes type values and attribute values of point and edge data; corresponding schemas are queried based on the type values of the graph data to determine a number of attribute byte groups corresponding to the type values, where the schemas are used to indicate attributes of the point and edge data, and each attribute byte group is used to store an attribute value of a corresponding attribute; convert the attribute values of the graph data into binary forms; and store the attribute values of the graph data that are represented in the binary forms-according to a byte group sequence format that matches the number of determined attribute byte groups.