126 results about "Small files" patented technology

The invention relates to the field of cold and hot data storage, in particular to a cold and hot data storage optimization method based on a large model, which comprises a business integration module, a data dependence module, a mode updating module, a cold and hot partitioning module and a file merging module, and is characterized in that the business integration module is used for collecting business data to form a training data set; the data dependency module is used for fitting business data to obtain a data consanguinity model, the mode updating module is used for predicting an access mode, the cold and hot partitioning module is used for calculating the heat of the data and performing dynamic scheduling, and the file merging module is used for automatically merging small files. A data storage structure is standardized, storage space is reduced, more efficient and more accurate cold and hot data classification can be achieved, the resource utilization rate of data storage hardware is improved, hot data response delay is reduced, the system storage utilization rate is remarkably improved, and the storage efficiency is improved.

The invention discloses an IO optimization method and device based on distributed storage, equipment and a storage medium, and relates to the technical field of distributed storage. Comprising the following steps: constructing a distributed metadata index layer between a client and a metadata server, and maintaining a dynamic mapping relationship between a path prefix and the metadata server, so that when the client initiates a file access request, a target metadata server can be directly positioned based on the mapping relationship; and whether the request is routed to the temporary metadata server for load balancing or not is dynamically determined according to the real-time load information of the temporary metadata server. Therefore, the problems of metadata request amplification caused by step-by-step analysis of a tree directory and non-uniform load of a metadata server caused by static directory binding in related technologies are solved; the technical effects that the metadata interaction delay is remarkably reduced, the throughput of the system in massive small file and burst IO scenes is improved, and a single metadata server is prevented from becoming a performance bottleneck are achieved.

Systems and methods are provided for creating a machine learning (ML) model staging repository, where ML models are stored as an open container image (OCI). The OCI comprises layers or portions of the complete ML model, so that the model can be stored separately and as a smaller files. The ML models may be pre-packaged for automated downloads and integration at the customer site. In some examples, the OCI can identify / store the model in a directory structure that defines the model and its profile. The OCI can comprise a combination of layers and profiles that allows the AI platform to optimize the storage and simplify the process of downloading or updating the given user namespace instead of downloading a single large file for the model.

The invention relates to a Json file parallel analysis and data writing method and device based on DataX. The method comprises the following steps: acquiring a source end Json file set, and dividing the source end Json file set into small file subsets and large file subsets according to file sizes; the small file aggregation path is configured with a json reader of the DataX, and parallel reading and analysis are carried out; splitting a root array of the large file into a plurality of independent small Json files through a Python script, aggregating paths, and reading and analyzing by using a DataX parallel channel; and finally executing the DataX Job to write the data into a target end. Through a classification processing strategy, the large-scale Json data analysis writing efficiency is remarkably improved, the large file memory occupation risk is reduced, the file scale is automatically adapted, a mature DataX frame and a Python script are reused, high performance and low maintenance cost are both considered, multi-type target end storage is supported, and the problems that a traditional tool is weak in parallel, high in resource consumption, poor in flexibility and the like are effectively solved.

The invention discloses a file access method, device and equipment and a computer readable storage medium, and belongs to the technical field of storage. The method is applied to a first node in a file system and comprises the steps that a first write request for a file is received, the first write request comprises first file data and IO information, and the IO information represents an address range for writing the first file data; in response to the first write request, metadata of the file is obtained, and the metadata comprises an extension attribute and a file size; according to the IO information and the file size, under the condition that it is determined that the written file size does not exceed the reference threshold value, the first file data is written into the extension attribute, and the written file size is the file size after the first file data is written. The file data is embedded in the metadata of the file, and the file data can be accessed while the metadata of the small file is accessed, so that the step of accessing the file data is reduced, the response time delay of the file system to the access request is reduced, and the purpose of improving the access performance of the system is achieved.

The invention discloses an IO merging mechanism based on a block index, relates to the technical field of big data and information retrieval, and optimizes an original multi-file stream output mechanism of open source Lucene into single-file stream output by expanding a Directory interface of the Lucene, so that IO operation merging is realized in an index writing process, and the operation efficiency is improved. The method specifically comprises three parts of a file storage structure, a writing process and a reading process. The file storage structure comprises a single file structure of a file header, a data area and a file tail; during writing, data is cached in the memory block firstly, and after the memory block is full, the data is added to a file by taking a data block as a unit; during reading, quick positioning is realized by analyzing the block index at the end of the file; according to the method, the problems of performance bottleneck, small file flooding, difficulty in atomicity guarantee and the like caused by multi-file IO in the prior art are solved, and the index construction efficiency and the system stability are remarkably improved.

The invention discloses a real-time storage optimization method based on block data, which relates to the technical field of big data processing and distributed storage, introduces a block data concept, realizes block-by-block writing, in-block ordering, block-level ordering and an intelligent flush trigger mechanism, and realizes real-time writing with high throughput, low delay and low memory occupation. The method specifically comprises the steps that block data is introduced into a data memory model to serve as a basic processing unit, multiple rows of data are aggregated into a block according to an aggregation key CKValue, row data in the block are sequenced according to a primary key PK, and write-in, namely aggregation is achieved; writing data, aggregating according to blocks, and uniformly sorting in the blocks; one block, namely multiple rows of data, generates an FRC file, and small files are reduced; exception processing: supporting block-level recovery to skip a failed block; the core problems of too high memory occupation, large CPU consumption, high writing delay and the like in a high-concurrency and high-throughput real-time writing scene can be solved, and the method is widely applied to scenes with extremely high requirements on data real-time performance and system stability.

The embodiment of the invention relates to a multi-system AI repository controller. Systems and methods are provided for creating a hierarchical repository of machine learning (ML) models in which ML models are stored as open container images (OCIs). The OCI includes a layer or portion of a complete ML model so that the model can be stored separately and as a smaller file. The ML model may be pre-packed for automated downloading and integration at the customer site. In some examples, the OCI may identify / store the model in a directory structure defining the model and its configuration files. The OCI may include a combination of layers and configuration files that allow the AI platform to optimize storage and simplify the process of downloading or updating a given user namespace rather than downloading a single large file for the model.

Adaptive content delivery systems and methods are disclosed herein. An example system is configured to determine capabilities of a requesting end user system, interface with a database designed to store images characterized by criteria, select an image from the database that balances a highest quality and a smallest file size for the end user system, based on the capabilities, and provide the image to a network for delivery to the end user system.

A method for simulating file system aging is disclosed, including: determining a storage capacity of a first device and a type of a file system; determining a ratio between a plurality of small files having different sizes based on the storage capacity of the first device and the type of the file system; copying the plurality of small files having different sizes to the first device according to the ratio; and deleting a part of the small files among all the small files copied onto the first device according to a predetermined interval pattern.

The invention discloses a cache optimization method and device based on distributed storage, and relates to the technical field of distributed storage systems. The method comprises the following steps: dynamically calculating a weight factor of a cache file, attenuating the weight factor based on the access frequency and timeliness of the file, and compensating and adjusting according to the size of the file; the maximum file threshold value set by the system is dynamically adjusted according to the size distribution of the files in the cache, and the maximum file threshold value is determined based on a preset quantile range so as to eliminate interference of extremely large files on weight calculation; when the weight factor of the cache file is lower than a preset threshold value, the file is removed from the cache, and the cache space is released; when the cache capacity is insufficient, the file with the minimum weight is selected for replacement based on the weight factor of the current cache file, and the high-value file is preferentially reserved in the cache. According to the method, the cache hit rate of small files in the distributed storage system is effectively increased, cache space waste and metadata operation overhead are reduced, and system performance and resource utilization rate are remarkably improved.

The invention discloses a method and device for fusing, storing and managing massive small files and data records, and the method comprises the steps: building an external metadatabase which is used for managing a hierarchical relation of scientific observation data sources and a mapping relation between an aggregation file and an original file of scientific observation data; according to a preset aggregation rule based on data product categories and timelines, the multiple original files are aggregated and packaged in a target file container in a column storage format, and the content of the original files and structured data records obtained by analyzing the original files are stored in the target file container at the same time; and based on the external metadatabase and the target file container, providing file-level access aiming at the original file content and record-level retrieval service aiming at the structured data record. According to the method, the aspects of system simplification, performance optimization, space saving and service capability are remarkably improved, and a new way with high expandability is provided for efficient storage and multi-level service of space weather scientific data.

The application provides a small file storage method, device, equipment, medium and program product. The method comprises the following steps: performing hash calculation on the storage path of a small file, determining a target large file bucket to which the small file is to be stored based on the obtained hash value, and the target large file bucket corresponding to a target metadata bucket and a target file name bucket; when the sum of the data amount of the small file and the data amount of the target large file bucket exceeds a preset bucket capacity threshold, generating a new large file bucket, a new metadata bucket and a new file name bucket under the same directory of the target large file bucket, and migrating the stored part of the small file, metadata and file name in the target large file bucket, the target metadata bucket and the target file name bucket to the new large file bucket, the new metadata bucket and the new file name bucket, so as to store the small file, the metadata thereof and the file name of the small file into the target large file bucket, the target metadata bucket and the target file name bucket respectively. The application can significantly improve the storage performance and reading efficiency of the small file.

The application discloses a kind of based on machine learning's genome analysis parallel optimization method and system, the method of the present application includes the input BAM file is cut into multiple same size file blocks;For each file block, extract file block characteristics, and input the file block characteristics into the pre-trained machine learning model to obtain the predicted running time of the file block;The predicted running time is in the front part file block is divided into smaller file block;File block is input into HaplotypeCaller in parallel to carry out variation detection;The variation detection result generated by HaplotypeCaller for each file block is merged and output.The present application is aimed at solving the problem of serious calculation tilt and low resource utilization caused by unclear calculation complexity before HaplotypeCaller runs, improving the efficiency of genome analysis, shortening the variation detection duration.

The application provides a voiceprint registration method, device, equipment and storage medium, the method comprises the following steps: storing the audio to be registered according to the time length of the audio to be registered when obtaining one audio to be registered every time; acquiring the first type of audio from the stored several audios based on a plurality of small file consumption service threads, and performing voiceprint registration on the first type of audio based on the operation equipment; and acquiring the second type of audio from the stored several audios based on a single large file consumption service thread, and performing voiceprint registration on the second type of audio based on the operation equipment; wherein the audio time length of the second type of audio is greater than the audio time length of the first type of audio. The voiceprint registration method provided by the application can perform voiceprint registration on short audios in a multi-thread concurrent manner (concentrate processing short audios), and perform voiceprint registration on long audios in a single thread, so that the resource utilization rate of the operation equipment can be effectively improved, the efficiency of voiceprint registration can be effectively improved, and the hardware cost can be greatly saved.

The present application discloses a fragment upload method, device and medium for a distributed storage system, which relates to the field of distributed storage, improves the upload efficiency of fragmented small files, receives a fragment upload request issued by a client, creates a fragmented small file, writes the fragment data of the fragment upload request in the fragmented small file, records the fragment number and fragment size information of the fragmented small file in a temporary index file, and after the fragment data upload is completed, creates an empty large file, the size of the large file is the actual size of the fragment data, reads the fragment information recorded in the temporary index file and updates it to the metadata of the large file. The uploaded fragment data is stored in the corresponding fragmented small file, and the fragment number and fragment size information of the corresponding fragmented small file are recorded in the metadata of the large file. There is no need to combine small files, which reduces the number of copies and improves the efficiency of fragment upload.

The invention provides a data processing method and system, a storage medium and a program product. The method comprises the steps of obtaining a to-be-uploaded target file; for a first file of which the file size is greater than a file size threshold value in the target file, uploading the first file to a search resource server by adopting a fragment uploading mode; and for the second file of which the file size is smaller than or equal to the file size threshold value in the target file, directly uploading the second file to the search resource server, and selecting a better uploading path for the files of different sizes through simple threshold value judgment. The small file avoids the overhead of multiple rounds of handshake and multiple requests in fragment uploading, and the small file is quickly uploaded directly through one request, so that the delay caused by network round-trip time is greatly reduced.

This application provides a backup method and an apparatus. The method includes: determining a first application that needs data backup; packaging and compressing a file whose size is less than a first threshold in a user space corresponding to the first application, to obtain a first compressed package, where the first compressed package is stored in the user space corresponding to the first application; and backing up the first compressed package from the user space corresponding to the first application to a user space corresponding to a second application. The method can reduce overheads caused by an I / O operation generated by backing up a large quantity of small-sized files between two user spaces, reduce a backup delay, and improve user experience.

The invention discloses a generation method and an analysis method for a data file of a question bank system. The method comprises the steps of generating a file overall structure; determining the position of a file header structure based on the overall file structure, and generating the file header structure based on the position of the file header structure; determining the position of a table index area based on the file header structure, generating a table index area structure based on the position of the table index area, determining the position of each metadata table in the table data area based on the table index area structure, and generating the table data area based on the position of each metadata table; determining the position of a file index area based on the file header structure, generating a file index area structure based on the position of the file index area, determining the position of each file resource in the file data area based on the file index area structure, and generating the file data area based on the position of each file resource. The technical problems that in the prior art, when a large number of small files are frequently and randomly accessed, decompression efficiency is low, memory occupation is high, and access delay is large are solved.

This application provides a method, apparatus, storage medium, and electronic device for storing small files. The method includes: obtaining a first small file identifier, wherein the first small file identifier is used to identify a first small file to be deleted; then deleting the first small file according to the first small file identifier and updating target index information; wherein the first small file is a small file packaged and stored in a first packaged file, the first packaged file includes multiple small files arranged in a preset order, the target index information includes the data length of the first small file; a second small file is a small file packaged and stored in the first packaged file and located after the first small file in a preset order; and then moving the second small file according to the updated target index information, wherein the distance moved by the second small file is the same as its data length. This application solves the problem of low efficiency in deleting small files in packaged files in related technologies.

The application relates to the technical field of computer application, and discloses a mass file data stream storage method, a storage system and a big data processing system. A rotation state conversion model is arranged between a data stream and a storage node, and storage pressure is dispersed through multi-node caching. The storage method comprises the following steps: presetting a file size demarcation point; inputting mass file data; judging whether the size of the input file is smaller than or equal to the demarcation point; storing file data smaller than or equal to the demarcation point in a selected storage node through the rotation state conversion model; generating index information of all small files, and merging the index information into an index file; organizing data in the form of a B+ tree for each index file to obtain an index data bucket; storing a plurality of index data buckets in the form of data blocks in the storage node; and saving all index data buckets in sequence as an index data bucket queue. The application can effectively solve the problems of traditional distributed storage systems, such as write performance decline and low efficiency when facing mass files.

The application provides a file transmission method, an electronic device and a storage medium, and relates to the technical field of intranet security protection. According to the write parameters of a to-be-written file, an index file operation interface is called to determine whether an index file of the to-be-written file exists, if yes, according to the operation state of the to-be-written file in the index file and the write parameters of the to-be-written file, it is determined whether the index file of the to-be-written file is updated, when the to-be-written file is a large file or a small file, the index file operation interface is called according to the total size of the to-be-written file to update the storage location information in the index file where the to-be-written file is located, the updated index file records the real storage location information of the to-be-written file, and the real storage location information of the to-be-written file has the characteristics of high utilization of the disk space occupied by different data files, the write efficiency of the to-be-written file is improved, and the high-performance transmission requirement of the one-way import system is met.

The invention relates to a cloud database computing cache system and an implementation method, and the system comprises a decentralized cache architecture module which is used for eliminating a centralized bottleneck, supporting ten-thousand-level concurrency and adapting to a big and small file mixed scene; the memory object sharing module is used for reducing intermediate result processing delay through intermediate result full-life-cycle zero-copy circulation and dynamic storage decision based on a memory pool and metadata service of the memory object sharing module; the self-adaptive asynchronous write-in module is used for connecting remote storage, multiplexing memory object sharing module resources and combining mechanisms such as delay confirmation, so that the write response speed is increased, remote requests are reduced, data integrity is guaranteed, and meanwhile, persistent support is provided for intermediate results. Through cooperative operation of the three modules, the cloud database computing cache system and the implementation method can effectively adapt to the requirements of high concurrency, large and small file mixing and elastic capacity expansion and contraction of a cloud native OLAP scene, and the integrity of data transmission and the stability of system operation are guaranteed.

This specification provides a request distribution processing method and a request distribution processing system. The request distribution processing method includes: obtaining file processing requests for target files; determining a target request processing queue set from multiple candidate request processing queue sets based on the request type of the file processing request, wherein the multiple candidate request processing queue sets correspond to different request types; determining a target request processing queue from the target request processing queue set based on the file identifier of the target file, and adding the file processing request to the target request processing queue; and invoking the request distribution resource corresponding to the target request processing queue to distribute and process the target requests in the target request processing queue. This eliminates the risk of read requests being starved due to resource contention, ensuring low-latency response for data loading; and ensures that processing requests for massive numbers of small files can obtain a fair scheduling opportunity, thereby preventing small files from being starved by large files.

The application provides a storage backup optimization method for long-term preservation of large-scale digital literature, relates to the technical field of data processing, and constructs an AIP storage path and a SUPPORT storage path by calculating the SHA512 hash value of a unique identifier of a digital object to be archived; encapsulates versioned core data files into an AIP directory according to the AIP storage path; stores and manages auxiliary data files into a SUPPORT directory according to the SUPPORT storage path, generates a disk slave copy, performs parallel integrity auditing on the disk master copy and the disk slave copy, performs consistency auditing and repair on the disk master copy and the disk slave copy, and outputs a repair completion state. The technical problems of low storage efficiency and long full-link backup period caused by massive small files in the prior art are solved. The technical effect of improving the storage efficiency and backup performance of massive small files is achieved.

The application discloses a one-key automatic signature method and system based on a CPE program small file, relates to the field of CPE programs, and comprises the following steps: receiving signature parameters input by a user through a graphical interface, including a signature branch, a key list, a storage capacity and a to-be-signed module path; dynamically matching the key list according to the signature branch, and converting the parameters into digital identifiers recognizable by a server end; after the to-be-signed file is standardizedly renamed, uploading the file to a Linux server through an SCP protocol, and calling a server signature interface to execute signature; after the signature succeeds, the system renames the file and automatically downloads the file to a local specified path. The system is composed of a parameter input module, a dynamic display module, a file processing module, a signature execution module and a security verification module, realizes the automation of small file signature, cross-platform compatibility and safe transmission, significantly simplifies the cumbersome process of small file burning and signature of CPE equipment, and improves signature efficiency and security.

The invention provides an atomization multi-resource data transmission optimization method. Metadata and a dependency relationship of the atomized resources are obtained by scanning project source codes, a resource dependency graph is constructed, and logic association and a hierarchical structure among the resources are accurately described; intelligent clustering and grouping are carried out based on the atlas in combination with resource use frequencies, and high-frequency co-occurrence and functional coupling resources are divided into the same resource group, so that the subjectivity and redundancy problems are avoided; the resource group is packaged into the set package file, and the list file is generated, so that aggregation packaging of the atomized resources is realized, and the integrity of resource delivery is ensured; in the application stage, client requests are intercepted, and corresponding set packages are loaded based on list file matching, so that the actual network request times are reduced, and the head overhead and the head-of-queue blocking risk are reduced. Therefore, not only is the performance bottleneck caused by a large number of small file requests solved, but also the on-demand loading and caching efficiency of the resources is guaranteed through a dynamic grouping mechanism, and the flexibility and execution efficiency of front-end resource transmission are fundamentally improved.

The invention provides a massive small file storage and real-time retrieval method and system based on Elasticsearch, which are used for solving the problems that the existing massive small file storage scheme cannot solve the problems of classification matching, real-time retrieval and the like of massive small files, or the offline deployment and maintenance cost is high and efficient bottom query interface support is lacked during retrieval. Or small file classification and query scenes need to be analyzed in advance, and certain universality is lacked. According to the Elasticsearch-based massive small file storage and real-time retrieval method provided by the invention, by adding boundary processing of the small files, metadata information and index information of the massive small files and file data are separately stored, and a plurality of small files are combined into a large file in a file storage system; therefore, the file storage system not only can efficiently store massive small files, but also supports massive storage of large files, and the compatibility of file storage is effectively enhanced.

The invention discloses a financial data merging method and device, equipment, a storage medium and a program product, and relates to the technical field of big data storage. In the application, a combined service cluster and a virtual node set are created, and the number of virtual nodes in the virtual node set corresponding to each combined service node in the combined service cluster is configured based on the load capacity index of each combined service node in the combined service cluster. Therefore, the mapping relation between the virtual node set and the merging service nodes in the merging service cluster is constructed, the merging service cluster and the virtual node set can be matched based on the mapping relation, and reasonable and balanced distribution of financial data small file merging tasks in a cluster environment is achieved. And furthermore, the merging task of the financial data small files is processed through the distributed target merging service node, so that high-performance and load-balanced financial data small file merging is realized.

The application discloses a front-rear end cooperative large PDF file intelligent generation method, relates to the technical field of computer software, and comprises the following steps: front-end rendering: the front end renders the work order data page by combining a headless browser with a Vue framework and an ElementUI component library, and dynamically displays structured work order details through a responsive data binding mechanism; data fragmentation: large work order data sets are split into N subtasks based on a dynamic fragmentation strategy; and small PDF files are generated in parallel: a containerized rendering cluster based on Kubernetes deployment management is used to allocate rendering tasks to each container; the small PDF files generated by the application through careful design with the help of the Vue and ElementUI frameworks are more attractive in terms of text layout, color matching and graphic display, and the large file after merging is more beautiful than the file generated by a traditional back end, thereby improving user experience.