Data Set Object Mode Switching in Dataflow Graphs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data management systems for graph-based data processing lack efficient methods for managing data set objects, particularly in creating associations between data set objects and nodes in dataflow graphs, which limits data transformation and processing capabilities.
Innovation Solution
A method for managing data set objects involves storing them in a data storage system and generating associations between data set objects and nodes in dataflow graphs, where each data set object has multiple modes with transformational logic for reading or writing data records, enabling flexible data processing and reuse across multiple graphs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data set objects are created separately for each node in dataflow graphs, then each node can have dedicated data transformation logic, but the system complexity increases and data processing efficiency decreases due to redundancy
Solution Approach 1:
The patent implements data set objects that can be universally reused across multiple nodes in dataflow graphs. Instead of creating separate data objects for each node, a single data set object can be associated with multiple nodes, enabling one object to serve multiple functions and reducing overall system complexity while maintaining dedicated transformation logic at each node through mode selection.
Solution Approach 2:
The patent merges multiple data set object instances into a single reusable template. By combining common data properties and transformation logic into a unified data set object that can be instantiated multiple times with different modes, the system reduces redundancy and improves processing efficiency while maintaining the ability to customize behavior at each node.
2Adaptability or versatility
If data set objects support multiple modes with different transformational logic, then data processing flexibility increases, but the object complexity and management difficulty increase
Solution Approach 1:
The patent implements dynamic mode switching within data set objects. Each data set object can change its transformational logic behavior based on the selected mode, allowing the object to adapt to different processing requirements. This dynamic capability enables a single object to replace multiple specialized objects while maintaining flexibility through runtime mode selection rather than static structure.
Solution Approach 2:
The patent segments the transformational logic into distinct modes within each data set object. By dividing the overall transformation logic into separate, manageable modes (such as read mode, write mode, different transformation types), the system makes complex objects more manageable and easier to configure while maintaining high versatility through mode combination.
3Loss of time
If data set objects are reused across multiple dataflow graphs, then development time decreases and consistency improves, but determining data dependencies becomes more complex
Solution Approach 1:
The patent implements feedback mechanisms that automatically track and report data dependencies when data set objects are reused across multiple dataflow graphs. The system monitors how data objects flow through different graphs and maintains dependency information, providing feedback to the system about data relationships. This automated tracking reduces the manual effort required to manage dependencies while enabling efficient data object reuse.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
Managing data set objects for graph-based data processing includes: storing a group (114) of one or more data set objects in a data storage system (116), the data set objects each representing a respective data set; and generating an association (210, 236) between at least a first data set object in the group and at least a first node of a dataflow graph for processing data in a data processing system, the first node representing a source or sink of data in a flow of data represented by a link in the dataflow graph, and the first data set object including a plurality of modes (402, 404, 406, 410, 412, 418) in which different transformational logic is applied to data processed by the first node.