28 results about "Dispersed storage" patented technology

The invention relates to the field of distributed storage, and discloses a distributed memory system based on erasure codes and a dynamic repair mechanism. Comprising a data node cluster, a metadata management cluster, a bandwidth token management module, a dynamic repair composer and a repair agent node. And the data node cluster divides the object into strip generation data fragments and verification fragments and dispersedly stores the data fragments and the verification fragments according to a placement rule. And the metadata management cluster maintains object stripe mapping, a fragment position set, erasure code protection configuration and a version epoch, and records a repair intention log. And the bandwidth token management module grants repair execution permission according to the fault domain budget. The dynamic repair composer generates availability repair and durability repair tasks, the temporary check fragments are written in when the number of the available check fragments is lower than a threshold value in the availability stage, and the configuration fragments are recovered and the temporary fragments are recovered in the durability stage. And the consistency of version epochs is verified before the agent node is repaired and written back, and if not, recalculation is carried out.

The invention belongs to the technical field of spaceflight electronic integrated circuits, and particularly relates to a satellite load FPGA configuration data error correction method and system capable of resisting single event upset, in particular to a bit-symbol double-layer error correction method of RS decoding operator error correction based on finite field algebra cascading on the basis of two-out-of-three majority voting logic. Single-particle independent upset and multi-copy common-mode failure can be corrected at the same time, and the problem of traditional TMR error locking is solved. Meanwhile, when configuration data are written into a nonvolatile memory, a cross-sector space staggered and dispersed storage mechanism is introduced, symbols in the same RS code word are separated as far as possible in physical positions, and therefore aggregated physical damage is converted into dispersed symbol-level errors.

A computing device for use in a storage network includes one or more network interfaces, memory with operational instructions and a processing module. The processing module is configured to execute the operational instructions receive a storage request for data and select a plurality of memory devices for storage of the data. The processing module partitions the data into a plurality of chunk sets and encode each chunk set of the plurality of chunk sets according to a dispersed storage error coded function to produce a plurality of chunks. Finally, the processing module facilitates storage of each chunk of the plurality of chunks in a memory device of the plurality of memory devices.

The invention discloses an HDFS (Hadoop Distributed File System) intelligent hierarchical storage method and system based on multi-dimensional popularity perception and tape library integration, and belongs to the technical field of distributed file systems, and the method comprises the following steps: expanding HDFS metadata to construct a unified namespace; acquiring data block access features and dynamically dividing hot, warm and cold data by using a K-means clustering algorithm; executing a self-adaptive grading strategy: adding a copy to hot data and storing in an SSD (Solid State Disk), maintaining a default copy to warm data and storing in a disk, and migrating cold data to a tape library and storing by adopting an erasure code; scattered storage and fault tolerance of relevance perception are realized through a message queue; metadata persistence and periodical re-evaluation are carried out, and automatic migration and dynamic adjustment of data among the SSD layer, the disk layer and the tape library layer are achieved. By adopting the method and the system, the storage strategy and the data access value are accurately matched, the storage cost is finally reduced, and the system I / O performance and the resource utilization rate are improved.

A method for a storage network computing device includes receiving a request to store a data object and generating a plurality of write slice requests, where the plurality of write slice requests includes a set of encoded data slices to be stored in the storage network and at least a decode threshold number of encoded data slices of the set of encoded data slices is required to recover a corresponding portion of a data object. The method continues by identifying a set of storage devices (from a plurality) for storage of the set of encoded data slices, where the plurality of storage devices includes one or more first memory devices local to the computing device and one or more second memory devices that are remote from the computing device. The method then continues, by storing the set of encoded data slices in the identified set of storage devices, generating a set of write slice responses relating to the set of encoded data slices and outputting the set of write slice responses.

A processing system of a storage network operates by receiving a write request associated with a data object; identifying dispersed storage error encoding parameters; and determining a number of available storage units. When the number of available storage units is determined to exceed a first number, the processing system operates by: determining a first write threshold number in accordance with the dispersed storage error encoding parameters; dispersed error encoding the data object to produce a first write threshold number of encoded data slices; and writing, to available storage units, the first write threshold number of encoded data slices. When the number of available storage units is determined to not exceed the first number, the processing system operates by: determining a second write threshold number in accordance with the dispersed storage error encoding parameters associated, wherein the second write threshold number is less than the first write threshold number; dispersed error encoding the data object to produce a second write threshold number of encoded data slices; and writing, to the available storage units, the second write threshold number of encoded data slices.

This invention relates to the field of distributed storage technology and provides a data collaboration method and system for distributed transaction terminals. The method involves each transaction terminal performing erasure coding on its local ledger data to generate a set of data fragments consisting of original data fragments and verification data fragments. These data fragments are then distributed and stored across different transaction terminals according to distributed storage rules, and the terminals exchange online status and fragment integrity information. When a request to access the target ledger is received, the terminal storing its fragments is located based on the ledger identifier, and the fragments are retrieved in parallel to reconstruct the target ledger. When a terminal is detected to be offline or a fragment is corrupted, online terminals collaborate to reconstruct the original data using surviving fragments, re-encode new fragments, and distribute them to healthy terminals. This invention, through erasure coding and distributed collaborative storage, reduces storage overhead while improving system fault tolerance and data availability, achieving efficient and reliable distributed ledger data collaborative management.

The invention discloses a data storage method and a related device, and the method comprises the steps: firstly obtaining factory data which needs to be stored through a TLC NAND technology, then compressing the factory data to obtain compressed data, dividing the compressed data into a plurality of strips, then generating a corresponding XOR check code for each strip, and storing the XOR check code in a database; and finally, the compressed data and the XOR check code corresponding to each strip are written into different physical units, so that the factory data are compressed and then put into a disk, the written data volume can be reduced, the abrasion of the embedded storage equipment can be relieved, the service life can be prolonged, and the storage efficiency can be improved by utilizing the XOR check code. The XOR check code and the survivor data are stored in different physical units in a dispersed manner, so that data recovery and reconstruction can be performed through the XOR check code and the survivor data when part of data in the strip is lost, the data integrity requirement of the embedded storage device for sudden power failure is met, the persistent retention capability of data storage is further improved, and the data storage efficiency is improved. And the reliability of data storage is improved.

The application relates to the technical field of computer application, and discloses a mass file data stream storage method, a storage system and a big data processing system. A rotation state conversion model is arranged between a data stream and a storage node, and storage pressure is dispersed through multi-node caching. The storage method comprises the following steps: presetting a file size demarcation point; inputting mass file data; judging whether the size of the input file is smaller than or equal to the demarcation point; storing file data smaller than or equal to the demarcation point in a selected storage node through the rotation state conversion model; generating index information of all small files, and merging the index information into an index file; organizing data in the form of a B+ tree for each index file to obtain an index data bucket; storing a plurality of index data buckets in the form of data blocks in the storage node; and saving all index data buckets in sequence as an index data bucket queue. The application can effectively solve the problems of traditional distributed storage systems, such as write performance decline and low efficiency when facing mass files.

One or more techniques and / or computing devices are provided for managing an arbitrary set of storage items using a granset. For example, a storage controller may host a plurality of storage items and / or logical unit numbers (LUNs). A subset of the storage items are grouped into a consistency group. A granset is created for tracking, managing, and / or providing access to the storage items within the consistency group. For example, the granset comprises application programming interfaces (APIs) and / or properties used to provide certain levels of access to the storage items (e.g., read access, write access, no access), redirect operations to access either data of an active file system or to a snapshot, fence certain operations (e.g., rename and delete operations), and / or other properties that apply to each storage item within the consistency group. Thus, the granset provides a persistent on-disk layout used to manage an arbitrary set of storage items.

The application belongs to the technical field of aerospace electronic integrated circuits, and particularly relates to a satellite load FPGA configuration data error correction method and system resisting single event upsets, a double-layer error correction method of "bit-symbol" based on RS decoding operator error correction of a finite field algebra on the basis of a "two out of three" majority voting logic, which can correct single particle independent upsets and multiple copy common mode failures at the same time, and solve the traditional TMR error locking problem. Meanwhile, when configuration data is written into a non-volatile memory, a cross-sector spatial staggered dispersion storage mechanism is introduced, so that the symbols in the same RS code word are separated as much as possible in the physical position, so as to convert the aggregation physical damage into dispersed symbol level errors.

A computing device of a storage network includes a network interface, memory and a processing module operably coupled to the memory and the network interface, with the processing module configured to execute the operational instructions to receive a storage request for a data file and then determine a data type for the data file. Based on the data type, the computing device determines whether to partition the data file into variable-sized data blocks and in response to a determination to do so, partitions the datafile according to a partition function into a plurality of data blocks, encodes each data block according to a dispersed storage error coded function to produce a plurality of data fragments and facilitates storage of the plurality of data fragments in a plurality of storage network nodes.

A computing system is operable to decode a plurality of incoming streams of digitally encoded data packets to extract a plurality of sets of input data from the plurality of incoming streams of digitally encoded data. A plurality of encrypted user data is generated based on encrypting at least some of the plurality of sets of input data and each of the plurality of encrypted user data is stored via a plurality of different storage locations. The plurality of sets of input data are processed generate a plurality of automatically generated textual data. A plurality of further outgoing streams of digitally encoded data packets are generated and transmitted via processing the plurality of automatically generated textual data.

The invention discloses a data collaborative storage management method and system for credit investigation data, and relates to the technical field of storage systems. The data collaborative storage management method and system for the credit investigation data comprises the steps that S1, the credit investigation data are collected and preprocessed, the credit investigation data are stored, and a credit investigation database is constructed; s2, carrying out consistency risk analysis through the fragmented data and the write-in operation data; s3, carrying out write-in orderliness analysis in combination with the global write-in sequence, the sorting priority and the consistency risk data; s4, repairing the fragmented data, and carrying out consistency repairing verification analysis based on the fragmented data before and after repairing; and S5, according to the fragmentation state, operating snapshots and repairing log data, archiving credit investigation data and continuously completing risk management and control. The problems that data are stored in a scattered mode, dirty writing conflicts occur frequently and the fragment consistency guarantee capacity is insufficient in the high-concurrency writing environment of the multi-service system are solved.

The utility model relates to the field of storage devices, in particular to an art tool integrated storage device which comprises a storage box and a storage assembly. A storage assembly used for classified storage according to art tools is installed in the storage box. The storage assembly comprises a sealing plate, a paint frame, a storage cabinet and a storage plate. The drawing board, tools and pigment are integrally and intensively stored through the pigment frame, the storage cabinet and the storage plate, limited space is fully utilized through reasonable layout, space waste caused by scattered storage is avoided, the disordered situation is reduced, the number of objects needing to be carried is reduced, follow-up movable creation is facilitated, and the drawing board is convenient to use. Meanwhile, all art tools are stored in a classified mode, a user can conveniently manage and search needed articles, the use convenience is improved, the use device is integrally stored, good sealing performance and protectiveness can be provided for the tools and the drawing board, paint is prevented from drying up or being affected with damp, the drawing board and the tools are protected against damage, and the overall storage effect is improved.

A method for execution by a computing device of a storage network includes appending at least a decode threshold number of encoded key slices of a set of encoded key slices to at least some encrypted data segments of a plurality of encrypted data segments to produce secure packages. The method further includes error encoding, in accordance with error encoding parameters, the secure packages to produce sets of encoded data slices, where a first secure package of the secure packages is dispersed storage error encoded using an error encoding function of the error encoding parameters to produce a first set of encoded data slices of the sets of encoded data slices. The method further includes outputting the sets of encoded data slices for storage in memory of the storage network.

This invention provides a method for transmitting railway Building Information Modeling (BIM) model files based on distributed storage. The method includes: lightweight decomposing the railway BIM model file into multiple sub-files; distributing these sub-files across edge server nodes using a cloud server; transmitting the sub-file list to a gateway; constructing a BIMdex index model; training the BIMdex index model using a training dataset; deploying the trained BIMdex index model locally on the gateway; and when an end device sends a file request carrying a file identifier to the gateway, the BIMdex index model in the gateway queries the sub-file list based on the file identifier, obtains the corresponding BIMdex address index, and returns it to the end device. This invention significantly improves data transmission efficiency by distributing sub-files of the same railway BIM model across multiple physical nodes and combining this with load balancing technology.

The invention relates to the technical field of civil aviation data security, in particular to a data security protection method and system for a civil aviation data intermediate station, and the method comprises the steps: obtaining first data; dividing the high-secrecy-level data into N data sub-blocks according to data types, and storing the N data sub-blocks in N storage modules in a dispersed manner; generating a confusion variable by adopting a random number generation algorithm, decomposing the confusion variable into N confusion sub-blocks, and storing the N confusion sub-blocks in N storage modules; destroying the confusion variables; generating a limited access strategy through a relevance blocking method; and when the formatting instruction is received, backing up the confusion sub-blocks stored in the storage module which receives the formatting instruction to the storage module which does not receive the formatting instruction, and then formatting. According to the method, the high-secrecy-level data is stored in a distributed manner, and the confusion sub-blocks are generated to confuse the relevance feature data, so that the data security is improved on the premise that the data value of the relevance feature data is brought into full play.

This application relates to a configuration file distribution method, apparatus, computer device, readable storage medium, and program product, falling under the field of network security technology. A connector in a second network generates an encrypted configuration file request based on a connector identifier. The configuration center determines the connector identifier and identity information based on the configuration file request, retrieves the target configuration file corresponding to the identity information from multiple configuration files, and configures the connector based on the target configuration file to handle financial transactions between the financial institution and platform to which the configuration center belongs. Compared to traditional data transmission over the public network, this solution, through a configuration center located in the first network, centrally manages the configuration files of connectors corresponding to various platforms. By authenticating the connector requesting the configuration file, it determines whether to distribute the corresponding configuration file, reducing the risk of arbitrary tampering due to dispersed storage of configuration files and improving the security of configuration file distribution.

The application relates to the technical field of data storage, in particular to a multi-engine intelligent data warehouse storage model construction method and system, which solves the technical problem that, in the prior art, a sharding random dispersion storage strategy is adopted for unstructured data, resulting in a large input / output overhead and high access cost in a data access calling process. The method comprises the following steps: constructing a distributed storage architecture comprising a plurality of storage nodes; performing sharding processing on unstructured data to be stored to obtain a plurality of data shards and acquire data features of each data shard; comparing the similarity of the data features between each data shard and historical data to evaluate the access granularity of each data shard; establishing an intelligent matching rule between the data shards and the storage nodes according to the access granularity of each data shard and the node resources of the storage nodes; and storing each data shard to a corresponding storage node according to the intelligent matching rule to form a multi-engine intelligent data warehouse storage model.

Methods and apparatus for maintaining availability of critical information in a storage network that includes a plurality of storage units. In an embodiment, a processing module(s) of the storage network identifies critical information stored in one or more memory devices of a computing device. Critical information can include, for example, information required for operation of one or more storage units to perform data access operations. The processing module obtains the critical information from the one or more memory devices, and dispersed storage error encodes the critical information to produce one or more sets of encoded critical slices. The processing module further selects a set of storage units of the storage network and facilitates storage of the one or more sets of encoded critical slices in the selected set of storage units.

The invention relates to the technical field of data storage, in particular to a multi-engine intelligent data warehouse storage model construction method and system, and solves the problems that in the prior art, a fragmentation random scattered storage strategy is adopted for unstructured data; and therefore, the technical problems of overlarge input and output overhead and relatively high access cost in a data access calling process are solved. The method comprises the following steps: constructing a distributed storage architecture comprising a plurality of storage nodes; performing fragmentation processing on the to-be-stored unstructured data to obtain a plurality of data fragments, and obtaining data features of each data fragment; assessing the access granularity of each data fragment by comparing the similarity of the data features between each data fragment and the historical data; establishing an intelligent matching rule between the data fragments and the storage nodes according to the access granularity of each data fragment and the node resources of the storage nodes; and storing each data fragment to a corresponding storage node according to an intelligent matching rule to form a multi-engine intelligent data warehouse storage model.

This invention relates to the field of civil aviation data security technology, specifically a data security protection method and system for a civil aviation data platform. The method includes acquiring first data; classifying high-confidentiality data according to data type. N Data sub-blocks, and stored in a distributed manner. N In each storage module; a random number generation algorithm is used to generate confusion variables, which are then decomposed into... N A scrambled sub-block, stored in N In each storage module, obfuscated variables are destroyed; a restricted access policy is generated using an association blocking method; upon receiving a formatting command, the obfuscated sub-blocks stored in the storage module that received the formatting command are first backed up to the storage modules that did not receive the formatting command before formatting. This invention improves data security by distributing high-security data and generating obfuscated sub-blocks to obfuscate association-related data, thereby fully leveraging the data value of association-related data.

The invention discloses a log processing method and device, and relates to the field of computing services. In the method, a plurality of logs are not located in the same storage directory, namely, the storage directory of at least one log is different from the storage directory of at least one of the other logs, so that a specified directory is configured, and reference files of the plurality of logs are created in the specified directory; the reference file of each log is used for storing the path of the log. Therefore, since the reference files of the plurality of logs are unified in the specified directory, the reference files of the plurality of logs can be quickly searched in the specified directory, so that the plurality of logs can be quickly acquired according to the reference files of the plurality of logs, and the dispersed stored logs can be mapped to the unified specified directory; the log collection path complexity is simplified, the log collection time is shortened, the cross-directory I / O operation frequency and resource consumption are reduced, and the overall log collection efficiency, real-time performance and timeliness are improved.

The embodiment of the application provides a data deletion method, device and equipment and a storage medium. The method first encrypts data and divides the data into blocks, converts the data ciphertext after the division into a binary form, and performs an inversion process, so that the public cloud does not store the real data ciphertext, and the ciphertext key is stored in the private cloud by using a shamir key sharing algorithm, self-key management is realized, a third-party auditing agency is not used, and the risk of data leakage is reduced. When a user wants to delete data, the user only needs to delete the key ciphertext corresponding to the data ciphertext in the main storage node. The application uses the key ciphertext deletion and the special data ciphertext processing mode to realize trusted deletion of cloud data, especially for a hybrid cloud architecture, security access of user data is guaranteed, and the security of data storage is improved.

A storage unit of a storage network is operable to determine rebuilding performance parameter values, where data is dispersed storage error encoded into a set of encoded data slices in accordance with error encoding parameters for storage via a set of storage units that includes the storage unit. A rate of internal rebuilding of encoded data slices is selected based on the rebuilding performance parameter values. At least one encoded data slice of the set of encoded data slices is internally rebuilt in accordance with the rate of internal rebuilding.

The invention discloses a data defragmentation method and device and a readable storage medium, and relates to the technical field of data storage. The data defragmentation method comprises the following steps: acquiring a data reading request from a data request end; obtaining correspondingly read target data according to the data reading request, determining the number of original logic blocks for storing the target data, and comparing the number of the original logic blocks with a number threshold value; in response to the fact that the number of the original logic blocks is larger than a number threshold value, recognizing an arrangement mark corresponding to the target data, and judging whether the target data is being subjected to data fragment arrangement or not according to the arrangement mark; and storing the target data in a continuous storage space in response to the situation that the data defragmentation is not executed. By implementing the embodiment of the invention, the data dispersedly stored in the discrete logic blocks can be stored in the storage units with continuous addresses, so that the number of the logic blocks occupied by the data is reduced, the data volume of metadata corresponding to the stored data is reduced, and the reading performance of the data is improved.

The invention relates to a cache management method and device, and the method comprises the steps: receiving a cache request instruction, receiving a new message corresponding to the cache request instruction when the cache request instruction is a storage instruction, and obtaining the partition storage space data of a shared cache; wherein the shared cache comprises a plurality of mutually independent cache partitions; the number of the cache partitions is greater than the read-write number corresponding to the cache request instruction; carrying out interleaving storage on the new messages according to the partition storage space data so as to enable the new messages to be stored in each cache partition in a scattered manner; and when the cache request instruction is a reading instruction, performing interleaving reading on the shared cache according to the cache request instruction and a secondary linked list corresponding to the shared cache to obtain a cache reading result. By adopting the method, a high-performance cache management method can be provided, a network burst packet can be absorbed, and a high-performance read-write service can be provided for an engine.