201 results about "Small files" patented technology

The invention discloses a solid state disk data reading method and system, a storage medium and a program product, and the method comprises the steps: calculating a temperature change rate and a temperature accumulation effect coefficient of a to-be-read data block according to a current temperature value and a historical temperature record; dividing the physical page address into a plurality of subintervals, and determining a loss level; calculating an initial reading voltage threshold value; compensating the initial reading voltage threshold value to obtain a target reading voltage threshold value; performing parallel reading on the to-be-read data block by adopting the target reading voltage threshold value to obtain original reading data; performing crosstalk correction on the original read data to obtain corrected data; when the reliability verification result of the corrected data of any subinterval is lower than a preset threshold value, error correction processing is carried out; and updating the loss level of the sub-interval according to the verification result of the reliability verification, and re-dividing the sub-interval of the data block to be read. The data reading delay is reduced, and the performance of the solid state disk in a scene of frequently reading small files is improved.

The invention discloses a small file distributed high-concurrency aggregation storage method and system, and the method comprises the steps: collecting performance index data, and transmitting the data to a resource coordination server; calculating a queuing rate; determining a target server according to the queuing rate, the file reading concurrency and the disk residual space; determining a first-level directory hash value; selecting a first-level storage directory and a second-level storage directory according to the first-level directory hash value; creating a target aggregation file in the secondary storage directory, generating a 2-byte file format field according to the compression identifier or the non-compression identifier, and generating a 4-byte file length field according to the size of the to-be-written file; writing the file format field, the file length field and the to-be-written file into a target aggregation file in sequence, and recording the initial offset of writing; and generating a file name containing the storage position information, and returning the file name to the client. According to the invention, the efficiency and effect of small file storage are optimized.

A method of data transfer between storage systems includes determining to transfer a dataset stored at a first storage system to a second storage system and invoking an operation to retrieve, for portions of the dataset having a size less than a threshold size, an identifier, metadata, and data stored in the portions of the dataset. The method further includes transmitting the identifiers, the metadata, and the data for the portions of the dataset having a size less than the threshold size to the second storage system.

The invention relates to the field of cold and hot data storage, in particular to a cold and hot data storage optimization method based on a large model, which comprises a business integration module, a data dependence module, a mode updating module, a cold and hot partitioning module and a file merging module, and is characterized in that the business integration module is used for collecting business data to form a training data set; the data dependency module is used for fitting business data to obtain a data consanguinity model, the mode updating module is used for predicting an access mode, the cold and hot partitioning module is used for calculating the heat of the data and performing dynamic scheduling, and the file merging module is used for automatically merging small files. A data storage structure is standardized, storage space is reduced, more efficient and more accurate cold and hot data classification can be achieved, the resource utilization rate of data storage hardware is improved, hot data response delay is reduced, the system storage utilization rate is remarkably improved, and the storage efficiency is improved.

The invention relates to the technical field of big data, and discloses a massive small file reading optimization method, system and device and a medium, and the method comprises the following steps: scanning small files in a distributed file system by using a Spark calculation engine to obtain metadata information of the small files; based on the metadata information, small files are classified, and the small files with the same type and data relevance are classified into one group; the resource states of cluster nodes are monitored in real time, the resource states comprise the processor utilization rate, the memory occupancy rate and the I / O load, the load score of each node is calculated, and small file task quotas are dynamically allocated; executing small file merging operation in parallel by utilizing a Spark memory calculation engine to generate a merged file containing a plurality of original small files; creating and maintaining index information in an external database, and recording the position and attribute of each original small file in the combined file; and providing a read-write interface through the special service layer, and positioning and accessing specific small file contents in the combined file.

The invention provides a data backup method, electronic equipment, a storage medium and a chip, and relates to the technical field of data backup, and the method comprises the following steps: during current backup, obtaining a file list during last backup, comparing a local file list during last backup with a local file list during current backup, finding out a newly added file and a modified file, and sending the newly added file and the modified file to the electronic equipment; the newly-added file and the modified file are uploaded to the cloud server for backup, and the same file is not repeatedly uploaded to the cloud server for backup any more; when a newly added file and a modified file are backed up, different processing modes can be executed aiming at a common file, a small file and a large file: the modified large file is segmented into a plurality of data blocks, the data blocks with differences are backed up, and the same data blocks are not backed up any more; the small files are uploaded in a packaging mode, if the reuse rate of the files in one package is larger than 90%, backup is not needed, and the files with the reuse rate smaller than 90% are repackaged and uploaded; according to the method, the backup time can be shortened, and the backup efficiency is improved.

The invention discloses a batch writing method and device of data and electronic equipment, the method is applied to the field of data storage, and the method comprises the following steps: receiving a plurality of writing requests; determining whether the data object belongs to a first type object or a second type object according to the storage space occupied by the data object in each write-in request; adding and writing at least one first type of object in the plurality of writing requests into a first data block in batches, and storing metadata information of the first type of object in the first data block; and slicing the second type of objects in the plurality of write requests, writing the second type of objects into a second data block, and storing metadata information of the second type of objects according to a mapping relationship between the sliced second type of objects and the second data block. Through the data writing method and device, the problems that in the process of processing the data writing request in the related technology, data of small files needs to be asynchronously gathered and deleted after being written, and large files occupy a large storage space due to the reference relation, so that the data writing efficiency is low are solved.

The invention discloses a data block processing method, system and device, a storage medium and a product, and relates to the technical field of data storage, and the method comprises the steps: obtaining a tree structure storage table, a leaf node storage table and a data block storage table, and when a node identifier matched with a first node identifier is found in each of the leaf node storage table and the data block storage table, sending a first node identifier to the leaf node storage table; establishing a first hierarchical relationship between the first node and the attribute information and the data block information of the first node; determining first small file list information and adding the first small file list information to the first hierarchical relationship to obtain a second hierarchical relationship; and generating a metadata snapshot according to the second hierarchical relationship and at least one third hierarchical relationship of other nodes, and sending the metadata snapshot to the client. According to the method, the metadata snapshot is issued to the client, so that the client directly reads the metadata snapshot from the local cache, the access pressure on a metadata management center is reduced, the metadata access efficiency is improved, and the problem of low supply efficiency of a small file data set in deep learning training operation is solved.

The invention discloses a distributed cloud native storage-oriented small file merging optimization method, which comprises the following steps of: converting data into a structure suitable for analysis by analyzing an HDFS (Hadoop Distributed File System) access log, and proposing a relevance analysis algorithm based on a user access mode. And discovering association rules from the file access records by using an FP-Growth algorithm: compressing transaction data and determining a frequent item set by constructing an FP tree, and generating the association rules conforming to the support degree and the confidence coefficient from the frequent item set. Respectively processing the small files with different relevance according to the generated rule; searching the small files with strong relevance, and merging the files by adopting a small file merging strategy based on a Huffman tree; small files with weak relevance are continuously added into a waiting queue, and when the queue meets the HDFS data node storage size condition, merging storage is carried out. According to the method, the file storage space of the HDFS is optimized, and the access efficiency of massive small files is improved. The method belongs to the field of correlation analysis and small file storage.

The invention discloses a method, a system and a medium for quickly uploading and downloading massive small files in batches, and belongs to the technical field of computer storage. The method comprises an initialization stage; an uploading preprocessing stage: preprocessing a to-be-uploaded file, and judging whether the to-be-uploaded file meets a batch uploading condition or not; in the intelligent packaging stage, intelligent packaging is carried out to obtain a packaged body; in the uploading execution stage, the packaged body is divided into a plurality of file blocks, and the file blocks are matched with hash values and then uploaded to an S3 object storage system; in the downloading initialization stage, after a client receives a file downloading request of a user, information of a to-be-downloaded file specified by the user is obtained, and file matching is carried out; in the downloading execution stage, the single file downloading process is executed for the single file downloading request; and executing a batch file downloading process for the batch file downloading request. The overall efficiency of data transmission is improved, the time required for uploading is shortened, and the method is particularly suitable for a large-scale data uploading scene.

The invention discloses an IO optimization method and device based on distributed storage, equipment and a storage medium, and relates to the technical field of distributed storage. Comprising the following steps: constructing a distributed metadata index layer between a client and a metadata server, and maintaining a dynamic mapping relationship between a path prefix and the metadata server, so that when the client initiates a file access request, a target metadata server can be directly positioned based on the mapping relationship; and whether the request is routed to the temporary metadata server for load balancing or not is dynamically determined according to the real-time load information of the temporary metadata server. Therefore, the problems of metadata request amplification caused by step-by-step analysis of a tree directory and non-uniform load of a metadata server caused by static directory binding in related technologies are solved; the technical effects that the metadata interaction delay is remarkably reduced, the throughput of the system in massive small file and burst IO scenes is improved, and a single metadata server is prevented from becoming a performance bottleneck are achieved.

The invention provides a large file transmission method and device, and the method comprises the steps: obtaining the uploading information of a currently uploaded target small file, the uploading information is used for indicating the uploading efficiency of the target small file, the target small file is at least one small file in a plurality of small files or a part of files in one small file, and the uploading information is used for indicating the uploading efficiency of the target small file; the small files are obtained by splitting a to-be-uploaded large file; according to the uploading efficiency indicated by the uploading information and the data volume of the target small file, a file corresponding to the target data volume is selected from the small files which are not uploaded for next uploading, and the uploading efficiency is in positive correlation with the difference value between the target data volume and the data volume of the target small file. According to the uploading efficiency of the small files, the small files which are not transmitted later are further split, integrated or maintained in the current situation, it can be ensured that the provided resources can be fully utilized by the small files transmitted every time later, the problem that transmission is too slow due to the fact that the data size of the files is too large does not occur, and the transmission efficiency of the large files is improved.

Systems and methods are provided for creating a machine learning (ML) model staging repository, where ML models are stored as an open container image (OCI). The OCI comprises layers or portions of the complete ML model, so that the model can be stored separately and as a smaller files. The ML models may be pre-packaged for automated downloads and integration at the customer site. In some examples, the OCI can identify / store the model in a directory structure that defines the model and its profile. The OCI can comprise a combination of layers and profiles that allows the AI platform to optimize the storage and simplify the process of downloading or updating the given user namespace instead of downloading a single large file for the model.

The invention is suitable for the technical field of small file processing, and provides an intelligent small file throughput optimization method and system. The method comprises the following steps: aggregating a plurality of small files into a logic file; a distributed metadata storage and indexing mechanism is adopted to store and index a plurality of small files; obtaining the access frequency and importance of the plurality of small files, and storing the plurality of small files in different storage media; and carrying out real-time load and access monitoring, dynamically adjusting the storage positions of the plurality of small files, and carrying out optimization adjustment on aggregation of the plurality of small files. The complexity and pressure of metadata management are remarkably reduced, and the storage efficiency and access performance of small files are fully improved, so that the bottleneck of the prior art is broken, the metadata volume can be effectively reduced, the utilization of the storage space is ingeniously optimized, and the access delay of the small files is greatly reduced. And the small file processing capability of the whole cluster storage system is improved.

The invention provides a distributed file processing method and device, equipment and a storage medium, relates to the technical field of computers, can be applied to various scenes such as cloud technology, artificial intelligence, intelligent traffic and auxiliary driving, and the method is applied to a distributed storage system and comprises the steps that in response to data back-brushing triggering for file fragments in a cache pool, the file fragments in the cache pool are cached; obtaining fragment information and fragment data volume of the file fragments, wherein the fragment information comprises a fragment sequence representing that the file fragments correspond to the file fragments in the to-be-processed file to which the file fragments belong; determining a data volume interval to which the file data volume of the to-be-processed file belongs based on the fragment information and the fragment data volume; if the belonging data volume interval is a small file interval, data processing is conducted on the to-be-processed file according to a preset multi-level back-flushing strategy, the small file interval comprises a plurality of sub-intervals, and the multi-level back-flushing strategy is used for indicating the mapping relation between the sub-intervals and various data operations corresponding to the data back-flushing trigger event. The access efficiency of the small file can be remarkably improved.

The invention relates to a Json file parallel analysis and data writing method and device based on DataX. The method comprises the following steps: acquiring a source end Json file set, and dividing the source end Json file set into small file subsets and large file subsets according to file sizes; the small file aggregation path is configured with a json reader of the DataX, and parallel reading and analysis are carried out; splitting a root array of the large file into a plurality of independent small Json files through a Python script, aggregating paths, and reading and analyzing by using a DataX parallel channel; and finally executing the DataX Job to write the data into a target end. Through a classification processing strategy, the large-scale Json data analysis writing efficiency is remarkably improved, the large file memory occupation risk is reduced, the file scale is automatically adapted, a mature DataX frame and a Python script are reused, high performance and low maintenance cost are both considered, multi-type target end storage is supported, and the problems that a traditional tool is weak in parallel, high in resource consumption, poor in flexibility and the like are effectively solved.

The invention discloses a file access method, device and equipment and a computer readable storage medium, and belongs to the technical field of storage. The method is applied to a first node in a file system and comprises the steps that a first write request for a file is received, the first write request comprises first file data and IO information, and the IO information represents an address range for writing the first file data; in response to the first write request, metadata of the file is obtained, and the metadata comprises an extension attribute and a file size; according to the IO information and the file size, under the condition that it is determined that the written file size does not exceed the reference threshold value, the first file data is written into the extension attribute, and the written file size is the file size after the first file data is written. The file data is embedded in the metadata of the file, and the file data can be accessed while the metadata of the small file is accessed, so that the step of accessing the file data is reduced, the response time delay of the file system to the access request is reduced, and the purpose of improving the access performance of the system is achieved.

The present application relates to the technical field of data backup. Provided are a data backup method, an electronic device, a storage medium, and a chip. In the method, during the current backup, a file list formed during the last backup is acquired, a local file list formed during the last backup is compared with a local file list formed during the current backup, newly added files and modified files are found, the newly added files and the modified files are uploaded to a cloud server for backup, and files which are the same are not repeatedly uploaded to the cloud server for backup any more. When the newly added files and the modified files are backed up, different processing modes can be executed in terms of ordinary files, small files and large files, wherein modified large files are segmented into a plurality of data blocks, data blocks which are different from each other are backed up, and data blocks which are the same are not backed up any more; and the small files are uploaded by means of packaging, and if a reuse rate of files in one packet is greater than 90%, no backup is required, and if the reuse rate is less than 90%, the files are repackaged and uploaded. The method can shorten the backup time and improve the backup efficiency.

The invention relates to a large-scale small file aggregation archiving storage and data addition, read-write and modification method, and the method comprises the steps: firstly obtaining a large-scale small file to be archived stored in a storage pool; and carrying out aggregation, filing and storage on the to-be-filed small files one by one based on the file aggregation structure capable of modifying the additional data. A file aggregation structure of modifiable appended data includes a metadata header field, an index field, a metadata block field, and a data block field. According to the large-scale small file aggregation archiving storage and data appending, read-write and modification method, a new file aggregation structure capable of modifying appended data is adopted to perform ordered aggregation archiving storage on the large-scale small files, and the structure allows direct modification and write-in operation on the small files in the aggregation file; and the whole file set does not need to be aggregated and archived again.

The invention discloses an IO merging mechanism based on a block index, relates to the technical field of big data and information retrieval, and optimizes an original multi-file stream output mechanism of open source Lucene into single-file stream output by expanding a Directory interface of the Lucene, so that IO operation merging is realized in an index writing process, and the operation efficiency is improved. The method specifically comprises three parts of a file storage structure, a writing process and a reading process. The file storage structure comprises a single file structure of a file header, a data area and a file tail; during writing, data is cached in the memory block firstly, and after the memory block is full, the data is added to a file by taking a data block as a unit; during reading, quick positioning is realized by analyzing the block index at the end of the file; according to the method, the problems of performance bottleneck, small file flooding, difficulty in atomicity guarantee and the like caused by multi-file IO in the prior art are solved, and the index construction efficiency and the system stability are remarkably improved.

The technology disclosed works in real time, as base and subordinate HTTP and HTTPS URL requests are received, to attribute subordinate requests to base web pages. The main case uses the “referer” or “referrer” HTTP header field for attribution, directly and through a referrer hierarchy to the base web page. A second case, which minimizes false generation of base web page log entries, involves small files, such as cascading style sheets (CSS) files, that often have a blank or no referrer field.

The technology disclosed works in real time, as base and subordinate HTTP and HTTPS URL requests are received, to attribute subordinate requests to base web pages. The main case uses the “referrer” or “referrer” HTTP header field for attribution, directly and through a referrer hierarchy to the base web page. A second case, which minimizes false generation of base web page log entries, involves small files, such as cascading style sheets (CSS) files, that often have a blank or no referrer field.

The invention discloses a small file rapid positioning and processing method and device for an HDFS distributed file system, electronic equipment and a storage medium, and the method comprises the steps: when it is monitored that a new fsimage file is generated, analyzing the new fsimage file into metadata, and sending the metadata to a database through a message queue, the metadata comprises metadata information about files and metadata information about folders; monitoring the metadata information about the file and the metadata information about the folder in the database, and positioning to the position of a small file; and according to the position of the small file, respectively executing corresponding treatment means on the data file and the metadata file. According to the method and the device, a'closed-loop 'strategy for quickly finding and managing the small files in the HDFS is realized, the stability of cluster operation can be improved, meanwhile, full-automatic execution can be realized, and the efficiency is greatly improved.

The invention discloses a real-time storage optimization method based on block data, which relates to the technical field of big data processing and distributed storage, introduces a block data concept, realizes block-by-block writing, in-block ordering, block-level ordering and an intelligent flush trigger mechanism, and realizes real-time writing with high throughput, low delay and low memory occupation. The method specifically comprises the steps that block data is introduced into a data memory model to serve as a basic processing unit, multiple rows of data are aggregated into a block according to an aggregation key CKValue, row data in the block are sequenced according to a primary key PK, and write-in, namely aggregation is achieved; writing data, aggregating according to blocks, and uniformly sorting in the blocks; one block, namely multiple rows of data, generates an FRC file, and small files are reduced; exception processing: supporting block-level recovery to skip a failed block; the core problems of too high memory occupation, large CPU consumption, high writing delay and the like in a high-concurrency and high-throughput real-time writing scene can be solved, and the method is widely applied to scenes with extremely high requirements on data real-time performance and system stability.

The embodiment of the invention relates to a multi-system AI repository controller. Systems and methods are provided for creating a hierarchical repository of machine learning (ML) models in which ML models are stored as open container images (OCIs). The OCI includes a layer or portion of a complete ML model so that the model can be stored separately and as a smaller file. The ML model may be pre-packed for automated downloading and integration at the customer site. In some examples, the OCI may identify / store the model in a directory structure defining the model and its configuration files. The OCI may include a combination of layers and configuration files that allow the AI platform to optimize storage and simplify the process of downloading or updating a given user namespace rather than downloading a single large file for the model.

Adaptive content delivery systems and methods are disclosed herein. An example system is configured to determine capabilities of a requesting end user system, interface with a database designed to store images characterized by criteria, select an image from the database that balances a highest quality and a smallest file size for the end user system, based on the capabilities, and provide the image to a network for delivery to the end user system.

A method for simulating file system aging is disclosed, including: determining a storage capacity of a first device and a type of a file system; determining a ratio between a plurality of small files having different sizes based on the storage capacity of the first device and the type of the file system; copying the plurality of small files having different sizes to the first device according to the ratio; and deleting a part of the small files among all the small files copied onto the first device according to a predetermined interval pattern.

The invention discloses a cache optimization method and device based on distributed storage, and relates to the technical field of distributed storage systems. The method comprises the following steps: dynamically calculating a weight factor of a cache file, attenuating the weight factor based on the access frequency and timeliness of the file, and compensating and adjusting according to the size of the file; the maximum file threshold value set by the system is dynamically adjusted according to the size distribution of the files in the cache, and the maximum file threshold value is determined based on a preset quantile range so as to eliminate interference of extremely large files on weight calculation; when the weight factor of the cache file is lower than a preset threshold value, the file is removed from the cache, and the cache space is released; when the cache capacity is insufficient, the file with the minimum weight is selected for replacement based on the weight factor of the current cache file, and the high-value file is preferentially reserved in the cache. According to the method, the cache hit rate of small files in the distributed storage system is effectively increased, cache space waste and metadata operation overhead are reduced, and system performance and resource utilization rate are remarkably improved.

The invention discloses a method and device for fusing, storing and managing massive small files and data records, and the method comprises the steps: building an external metadatabase which is used for managing a hierarchical relation of scientific observation data sources and a mapping relation between an aggregation file and an original file of scientific observation data; according to a preset aggregation rule based on data product categories and timelines, the multiple original files are aggregated and packaged in a target file container in a column storage format, and the content of the original files and structured data records obtained by analyzing the original files are stored in the target file container at the same time; and based on the external metadatabase and the target file container, providing file-level access aiming at the original file content and record-level retrieval service aiming at the structured data record. According to the method, the aspects of system simplification, performance optimization, space saving and service capability are remarkably improved, and a new way with high expandability is provided for efficient storage and multi-level service of space weather scientific data.

The application provides a small file storage method, device, equipment, medium and program product. The method comprises the following steps: performing hash calculation on the storage path of a small file, determining a target large file bucket to which the small file is to be stored based on the obtained hash value, and the target large file bucket corresponding to a target metadata bucket and a target file name bucket; when the sum of the data amount of the small file and the data amount of the target large file bucket exceeds a preset bucket capacity threshold, generating a new large file bucket, a new metadata bucket and a new file name bucket under the same directory of the target large file bucket, and migrating the stored part of the small file, metadata and file name in the target large file bucket, the target metadata bucket and the target file name bucket to the new large file bucket, the new metadata bucket and the new file name bucket, so as to store the small file, the metadata thereof and the file name of the small file into the target large file bucket, the target metadata bucket and the target file name bucket respectively. The application can significantly improve the storage performance and reading efficiency of the small file.