Object Storage Security Database Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing object storage systems face challenges in supporting low-latency key-value point, range, and prefix queries, especially when dealing with large volumes of data, due to high request latencies and lower bandwidth compared to local filesystems.
Innovation Solution
A system is developed that includes a security database with a manifest and a distributed index, where each security record has a key and a value, and the keyspace is partitioned into key ranges. This system uses an object storage system to store the manifest and segments of the distributed index, allowing for efficient querying and rating generation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If object storage systems are used to store large volumes of data, then storage capacity and reliability are improved, but query latency increases and bandwidth decreases
Solution Approach 1:
The patent segments the database into multiple partitions, each stored as a separate object in the object storage system. Each partition contains a subset of key-value pairs organized by key range. This segmentation allows the system to leverage the high capacity and reliability of object storage while enabling efficient querying by retrieving only the relevant partition(s) containing the requested keys, thus reducing query latency and improving bandwidth utilization.
2Quantity of substance
If object storage systems are used to store large volumes of data, then storage capacity and reliability are improved, but query throughput decreases
Solution Approach 1:
By dividing the database into multiple independent partitions stored as separate objects, the system enables parallel retrieval of multiple partitions simultaneously. This segmentation strategy allows high query throughput by conducting concurrent object requests to different partitions, effectively overcoming the throughput limitations of traditional object storage systems while maintaining large-scale storage capacity.
Solution Approach 2:
The patent introduces a partitioning dimension to the traditional object storage model, organizing data not just by key but by key ranges mapped to specific partitions. This dimensional change enables the system to retrieve data in parallel across multiple partitions, significantly improving query throughput while preserving the high-capacity storage benefits of object storage systems.
3Loss of time
If key-value databases are implemented using local filesystems, then query latency is reduced, but storage capacity and scalability are limited
Solution Approach 1:
The patent uses the object storage system as an intermediary layer between the database logic and physical storage. By implementing a database layer that translates queries into targeted object retrieval operations, the system achieves both low query latency (by retrieving only needed partitions) and high storage capacity (by leveraging cloud object storage), effectively mediating between the conflicting requirements of speed and scale.
Data Source
AI summary
A system for determining an entity's security rating may include a ratings engine and a security database. The security database may include a manifest and a distributed index containing security records. Each of the security records may have a key (e.g., a network identifier of a network asset) and a value (e.g., security information associated with the network asset identified by the key). The keyspace may be partitioned into multiple key ranges. The manifest may contain references to segments of the distributed index. Each segment may be associated with a key range and may index a group of security records having keys within the key range. The manifest and the segments may be stored in an object storage system. The ratings engine may determine the security rating of an entity based on security records of the entity's network assets, which may be retrieved from the database.