Heterogeneous Sensor Data Storage via Unified API and Caching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage architectures fail to efficiently manage and access semi-structured, heterogeneously-typed sensor data in a distributed environment, particularly for autonomous driving vehicles, due to challenges in security, efficiency, and ease of use, as they are not designed to handle the unique characteristics of sensor data such as varying data types, sizes, and rates, and lack flexibility and consistency.
Innovation Solution
A data storage access system that includes a design paradigm for representing sensor data, software-implemented routines for querying and joining data from different sensors, tools for importing and transferring data across geographically disparate regions, and a system using HTTP requests and caching to provide secure, efficient access to data stored in a network-attached storage system, integrated with authentication tools like Kerberos and IPA, and supporting time-series alignment and machine learning processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If conventional distributed file systems (HDFS, NFS) are used to access NAS, then data access simplicity is improved, but security and efficiency are compromised
Solution Approach 1:
The patent introduces an intermediary authentication and authorization layer between the distributed file system and NAS. This intermediary validates user credentials, enforces access policies, and logs security events, thereby improving security without compromising the simplicity of data access for authorized users.
2Ease of operation
If conventional distributed file systems (HDFS, NFS) are used to access NAS, then data access simplicity is improved, but access efficiency deteriorates
Solution Approach 1:
The patent implements preliminary actions by pre-authenticating users, pre-authorizing access paths, and pre-caching frequently accessed metadata before actual data access occurs. This reduces the overhead during data access operations, improving efficiency while maintaining simplicity through the established access framework.
3Volume of stationary object
If data is stored across multiple devices in NAS, then storage capacity is improved, but access complexity increases
Solution Approach 1:
The patent creates a universal access interface that works consistently across all storage devices in the NAS. This unified interface abstracts the underlying complexity of distributed storage, allowing users to access data from any device through the same simple mechanism, thereby maintaining ease of use while leveraging expanded storage capacity.
4Reliability
If authentication and logging are moved to a remote trusted server, then security is improved, but network dependency increases
Solution Approach 1:
The patent performs authentication and authorization actions in advance, caching the results locally on the distributed file system. This preliminary authentication reduces ongoing network dependencies, as subsequent access decisions can be made using cached credentials and policies, thereby maintaining high security while reducing network dependency for routine operations.
Data Source
AI summary
Various embodiments for implementing a system and method for managing and querying semi-structured, heterogeneously-typed sensor data in a distributed environment are disclosed. Example embodiments include: enabling data communications between a plurality of compute nodes and a plurality of physical data storage devices via an application programming interface (API) layer, a cache management layer, a server layer, and a storage layer; and receiving a data request from at least one of the plurality of compute nodes at the API layer, the data request including an identification of a topic of a dataset, the topic including a metadata file, a data file, and an index file, the index file including at least one pointer into the data file.