Multi-Tenant Analytics Pipelines With Metadata-Driven Tenancy Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data analytics platforms are limited by their architecture, requiring technical expertise, segregating data, lacking scalability and security, and failing to provide integrated insights across multiple data products.
Innovation Solution
A serverless data analytics platform that automates deployment and scaling, uses metadata-driven flows, and supports flexible access controls to enable less-technical users to process and analyze data across multiple data products securely.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If existing data analytics platforms use traditional architecture with physical infrastructure, then data processing can be performed, but scalability is limited and resource requirements are high
Solution Approach 1:
The patent replaces physical infrastructure (mechanical system) with cloud-based serverless architecture. The system uses cloud computing resources instead of on-premises server farms, allowing automatic scaling without physical hardware constraints. This substitution enables the platform to scale dynamically based on demand while maintaining data processing capabilities.
Solution Approach 2:
The patent implements a unified data platform that handles multiple data products and analytics workloads through a single infrastructure. The serverless architecture provides universal access to computing and storage resources for various analytics tasks, eliminating the need for separate physical infrastructure for each data product and enabling seamless scaling across different workloads.
2Productivity
If existing data analytics platforms require technical specialists for operation, then complex data processing tasks can be performed, but ease of operation deteriorates and requires multiple years of training
Solution Approach 1:
The patent implements self-service capabilities through automated metadata-driven pipeline creation. The system automatically generates data processing pipelines based on metadata specifications without requiring manual configuration by technical specialists. This automation allows business users to perform complex data processing tasks by simply defining high-level requirements, eliminating the need for extensive technical training while maintaining processing capabilities.
Solution Approach 2:
The patent introduces metadata as an intermediary layer between users and the complex underlying data processing infrastructure. Users interact with simplified metadata definitions rather than directly configuring complex pipelines. This intermediary abstraction shields users from technical complexity while enabling powerful data processing through automated pipeline generation from metadata specifications.
3Ease of manufacture
If existing data analytics platforms segregate data by data product, then data management is simplified, but the ability to gain integrated insights across multiple data products is lost
Solution Approach 1:
The patent merges multiple segregated data products into a unified data platform where all data resides in a common data lake. This consolidation allows the system to maintain simple data management through standardized processes while simultaneously enabling integrated analytics across all data products. The serverless architecture processes queries across the entire unified dataset, generating insights that span multiple original data products.
Solution Approach 2:
The patent implements logical segmentation through metadata tags and data lineage tracking rather than physical segregation. Data from different products is physically unified in the data lake but logically organized through metadata. This approach maintains the simplicity of unified data management while preserving the ability to distinguish and analyze data by its original source through metadata-based segmentation.
4Ease of operation
If existing data analytics platforms lack automation, then manual control over data extraction and loading is maintained, but technical support requirements increase and operational efficiency decreases
Solution Approach 1:
The patent implements self-service automation where the system automatically performs data extraction, transformation, and loading based on metadata specifications. The automated pipeline generation and execution eliminates the need for manual technical intervention in routine data management tasks. The system serves itself by automatically provisioning resources, managing data flows, and optimizing processing based on defined requirements, thereby increasing automation while maintaining operational control through metadata definitions.
Data Source
AI summary
A data analytics system configured to perform operations is disclosed. The operations can include creating, in response to instructions received from a user, a first pipeline. This pipeline can be configured to extract data from an append-only first data store, extract identifying characteristics from the extracted data, provide the identifying characteristics to an identity service, and receive a tenancy identifier from the identity service. The pipeline can further be configured to create a data object in a second data store using the extracted data; create a tenancy object in a metadata store, the tenancy object associated with the data object, the metadata store implementing a hierarchical data object ownership graph; and associate the tenancy object with a parent object in the hierarchical data object ownership graph. The data analytics system can then tear down the first pipeline.