16 results about "Hybrid storage system" patented technology

The present application relates to a big data hybrid storage system and method, and a readable storage medium. The big data hybrid storage system comprises: a first file system, which is used for storing active data; a second file system based on object storage, which is used for storing inactive data; a hybrid storage protocol endpoint, which is used for performing automatic data routing and reading / writing on the first file system and the second file system; and a data synchronization server, which is used for parsing an audit log of the first file system to obtain file usage statistical information, starting a timing task to scan the first file system and the second file system to obtain a corresponding first file list and second file list, and determining, on the basis of the file usage statistical information and the difference between the first file list and the second file list, whether to mirror and synchronize data in the first file system to the second file system. Thus, data storage costs are reduced, and object storage access pressure is alleviated.

Embodiments of the present disclosure are directed to a method for storing and processing data. The method includes identifying a database in a memory of a host device having one or more rows and one or more columns. A partition having a partition size is identified, and the one or more rows of the database is identified based on the partition size. The data stored in the one or more rows is converted into a column-based format, and the data is stored in a computational storage device in the column-based format. The computational storage device is configured to retrieve the data stored in the column-based format, in response to a query, and process the query based on the data.

This application provides a resource scheduling method, apparatus, electronic device, and storage medium for a hybrid storage system. The method includes: acquiring a set of files to be scheduled in a hybrid storage system, wherein the hybrid storage system includes hard disk drives (HDDs) and solid-state drives (SSDs); for each file to be scheduled in the set, determining the random access frequency, access popularity, file size, and read operation ratio of the file to be scheduled in the current period, wherein the read operation ratio is used to characterize the proportion of read operations to the total access volume; determining the file score corresponding to the file to be scheduled based on the random access frequency, access popularity, file size, and read operation ratio; and storing all files to be scheduled to the corresponding storage location according to the file score of each file to be scheduled. This achieves both reduced storage device costs and improved data access speed of the storage device.

The application provides a unified metadata layer hybrid storage method and system, the method comprises receiving a storage access request of each node, parsing the protocol type and the target data path corresponding to the request; based on the preset directory table, the index node table and the object table, the metadata operation corresponding to the protocol type is executed; the metadata operation is encapsulated as an atomic transaction submission, and the corresponding records of the directory table, the index node table and the object table are synchronously updated, and the read and write operations of the target data are completed. The application not only solves the problems of semantic fragmentation, poor consistency and performance bottleneck existing in the traditional hybrid storage system for a long time, but also solves the problems of resource contention, business insecurity and poor cluster stability in the multi-training task parallel scene according to the special needs of the AI large model training scene.

The present application relates to the technical field of coal mine safety, and discloses a method for constructing a knowledge graph in the field of coal mine disaster risk monitoring and early warning, which comprises collecting physical perception data, managing business data and professional knowledge data to form an initial data set; cleaning and standardizing the initial data set, and classifying and extracting characteristic attributes; using a deep learning model and a rule base to perform graph element extraction and fusion, and constructing a complete knowledge graph; constructing a hybrid storage system that cooperates with a graph database and a relational database, and performing joint queries; monitoring incremental data for the hybrid storage system, performing update operations, and completing adaptive dynamic evolution. The present application determines entity labels by using a dynamic prototype network, distinguishes explicit and implicit attributes based on a graph attention network model, and introduces a causal mask matrix to aggregate spatiotemporal causal correlation information, thereby solving the problems of entity hierarchical nesting and complex semantic correlation, and improving the accuracy and completeness of the knowledge graph.

The present invention comprises solar tubes and other components arranged in a repetitive manner to form solar bricks. More specifically, the solar bricks (102) are assembled to form a solar panel that permits multiple direct and indirect reflection of solar radiation to achieve multiple reflection due to induced optical whirl of solar rays of visible spectrum that minimizes reflectance loss. The assembly (100) includes a 3rd dimension arrangement of solar cell in solar tube that allows to trap solar radiation and utilise higher solar cell area w.r.t horizontal earth surface. The assembly includes Peltier cells suitable to operate at low differential temperature for harnessing and used solar energy that is converted to heat. Further, solar thermal and full spectrum photovoltaic generation is combined to derive maximum efficiency.

The application belongs to the technical field of electric power data security and data sharing, and discloses an electric power data security sharing method based on privacy calculation and related equipment. The hybrid storage system solves the problem of limited storage capacity of the blockchain. The on-chain storage metadata guarantees traceability, and the off-chain storage encrypted data improves storage efficiency. The improved BFT consensus algorithm guarantees the consistency of the distributed nodes and solves the trust problem of cross-organization data sharing. The collaborative computing protocol of secret sharing and homomorphic encryption takes into account the privacy protection strength and the computing and communication efficiency. Compared with the traditional single privacy calculation scheme, the computing efficiency is improved, the communication cost is reduced, the proxy re-encryption technology realizes controllable sharing of data, and the compliance requirements are met. The application can effectively support sensitive data sharing scenes such as electric power load prediction and cross-department collaborative analysis, and can guarantee that sensitive information such as power grid topology and user power consumption behavior is not leaked.

The application discloses a data storage method, controller and system for NVM and SSD, and belongs to the field of computer system structure, and comprises: static address mapping: a unified linear logical address space is provided externally, and is divided into an NVM-zone region mapped to an NVM device and an SSD-zone region mapped to an SSD device; metadata and file data are respectively mapped to the NVM-zone and the SSD-zone; dynamic request scheduling: a space is divided from the NVM device to serve as an NVM-buffer, and for file data mapped to the SSD-zone, if the system is not currently in a small write-intensive load scene, and the request is a small write request, the data is cached into the NVM-buffer, and a target address and a cache address of the data in the SSD device are recorded; otherwise, the data is directly written into the SSD device. The application can maximize the performance advantages of the NVM and the SSD device, and effectively improve the overall performance of the hybrid storage system.

The present disclosure provides methods, systems, devices, and media for data processing. The method may include obtaining information of one or more hard disks of each of one or more data nodes, the information of the one or more hard disks at least including type information, capacity information, and location information of each of the one or more hard disks; determining, based on a data processing instruction and the information of the one or more hard disks, target information of a target hard disk among the one or more hard disks; and transmitting the data processing instruction including the target information of the target hard disk to a target data node corresponding to the target hard disk.

The invention provides a self-adaptive hybrid storage system, which belongs to the field of computers, and comprises a request buffer area used for receiving a user request and temporarily storing write-in data in the user request; the high-performance solid state disk dynamically calculates and limits the user write-in bandwidth through a self-adaptive smoothing mechanism according to the write cache space of the high-performance solid state disk and the data migration speed of a data compaction module, and caches write-in data from a request buffer area; the data compaction module is connected between the high-performance solid state disk and the high-density solid state disk and is used for realizing migration of written data so as to release a write cache space; and the high-density solid state disk receives and finally stores the migrated write-in data, executes garbage collection operation based on the data migration write-in bandwidth and the storage space of the high-density solid state disk, and releases a new available storage space. According to the method, the obvious fluctuation of the size of the available space of the write buffer area in the high-performance solid state disk is effectively avoided, and the reduction of the user write-in performance caused by garbage collection under the sudden load is prevented.

The present application relates to a management method and management apparatus for a hybrid memory system, a device, and a medium. The method includes: acquiring a write hotness value and an access hotness value of a currently accessed memory page; comparing the write hotness value with a specified write hotness threshold, generating write hotness page migration flag information; comparing the access hotness value with a specified access hotness threshold, generating access hotness page migration flag information; grouping a hybrid memory, generating memory page management information corresponding to the hybrid memory based on memory page information of all grouped memory banks; determining whether the currently accessed memory page is a hot data page based on the write hotness page migration flag information, the access hotness page migration flag information, and current memory page management information, performing data migration based on a preset cold and hot data page migration strategy.

The adaptive hybrid storage system belongs to the field of computers and comprises a request buffer, a high-performance solid state disk, a data compaction module and a high-density solid state disk.The request buffer is used for receiving user requests and temporarily storing write data in the user requests.The high-performance solid state disk dynamically calculates and limits the user write bandwidth through an adaptive smoothing mechanism according to its own write cache space and the data migration speed of the data compaction module, and buffers the write data from the request buffer.The data compaction module is connected between the high-performance solid state disk and the high-density solid state disk, and realizes the migration of the write data to release the write cache space.The high-density solid state disk receives and finally stores the migrated write data, performs a garbage collection operation based on the data migration write bandwidth and its own storage space, and releases new available storage space.The application effectively avoids the significant fluctuation of the available space size of the write buffer in the high-performance solid state disk, and prevents the decline of the user write performance caused by garbage collection under a burst load.

The application provides a hybrid storage system, method and medium, comprising a client, a data interaction module, a file system adapter and an underlying storage system; the client accesses user requests in real time, parses the user requests and extracts user operation request sentences; the data interaction module converts the operation request sentences into operation sentences of the corresponding underlying storage system through the file system adapter, performs operations and returns operation results to the client; and the client returns the operation results to the user. The application realizes free conversion of a file system tree structure to a key value mode of an object storage system; and unstructured data such as audio and video and structured data under the file system can be directly accessed without the need for business code adaptation.

The present invention comprises solar tubes and other components arranged in a repetitive manner to form solar bricks. More specifically, the solar bricks (102) are assembled to form a solar panel that permits multiple direct and indirect reflection of solar radiation to achieve multiple reflection due to induced optical whirl of solar rays of visible spectrum that minimizes reflectance loss. The assembly (100) includes a 3rd dimension arrangement of solar cell in solar tube that allows to trap solar radiation and utilise higher solar cell area w.r.t horizontal earth surface. The assembly includes Peltier cells suitable to operate at low differential temperature for harnessing and used solar energy that is converted to heat. Further, solar thermal and full spectrum photovoltaic generation is combined to derive maximum efficiency.

The invention provides a navigation laboratory multi-mode data storage and efficient query method, and relates to the technical field of data storage and application, and the method comprises the following steps: adopting mixed storage combining a database and a file system; realizing recording of the database by utilizing distributed transaction management; establishing a combined index for the structured data in the database, and constructing a content index for the unstructured file in the file system by adopting Lucene; introducing a Redis cache technology, and performing cache replacement on the data by setting an LFU elimination algorithm in combination with a data thermodynamic value and a first-in first-out rule; loading the data to a cache step by step based on a historical access log and a data priority rule, and realizing quick duplicate judgment of the memory data through a Bloom filter; and querying an intermediate result according to the cache, and shunting the intermediate result at the same time. According to the method, the multi-mode data storage efficiency is optimized through a hybrid storage architecture and a loop log mechanism, and the consistency of cross-medium data is guaranteed by means of distributed transaction management.

The invention provides a mixed storage method and system for a unified metadata layer. The method comprises the steps that a storage access request of each node is received, and a protocol type and a target data path corresponding to the request are analyzed; based on a preset directory table, an index node table and an object table, executing metadata operation corresponding to the protocol type; according to the method, the problems of semantic segmentation, poor consistency, performance bottleneck and the like which exist in a traditional mixed storage system for a long time are solved, and the method and the device have the advantages that the method and the device aim at the exclusive requirement of an AI large model training scene, so that the method and the device are easy to implement. The problems of resource scrambling, unguaranteed service and poor cluster stability in a multi-training task parallel scene are solved.