32 results about "File allocation" patented technology

A storage system includes a storage device and a host device. The storage device includes a non-volatile memory device configured to store data and a storage controller configured to control the non-volatile memory device to isolate and store the data by grouping storage spaces of the non-volatile memory device into a plurality of isolated storage spaces. The host device includes a file system configured to generate and manage a plurality of file allocation instances corresponding to a plurality of file allocation system calls from an application program by allocating, with respect to each of the plurality of file allocation system calls, a logical address range and an isolated identifier of a plurality of isolated identifiers corresponding to the plurality of isolated storage spaces.

Systems, apparatuses, methods, and computer program products are disclosed for linking remote file storage. An example method includes identifying, by discovery agent circuitry, a set of files for storage from a set of candidate files, and classifying, by file metadata circuitry, each file from the set of files for storage into file usage categories. The example method also includes assigning, by file metadata circuitry, permission categories to each file from the set of files for storage based on the file usage categories, and providing, by communications hardware, the set of files for storage and associated metadata to a file storage host, wherein the associated metadata comprises the file usage categories and the permission categories The example method also includes updating, by the discovery agent circuitry, content stored on the file storage host, the content comprising the set of files for storage.

The invention discloses a file processing method and device, equipment, a storage medium and a computer program product, and the method comprises the steps: obtaining the attribute information of a file in response to a file processing request, and determining the to-be-read data volume of the file; performing priority ranking on the files based on the attribute information and the to-be-read data volume to obtain a ranking result; distributing the file to a first queue or a second queue according to the sorting result, the attribute information and the current processing capability; wherein the first queue is configured to instantly process the file, and the second queue is configured to delay processing of the file. Through the design of a flow pool system and double-queue parallel processing, the system load can be effectively balanced, resource waste is avoided, and quick response of file processing is ensured. Besides, the automatic length adjustment of the processing queue enables the system to better adapt to the fluctuation of the business volume, and the system can keep stable operation, reduce the failure rate and improve the user experience no matter whether the peak or the valley is processed.

In one embodiment, an illustrative method herein comprises: obtaining, by a process, a file having a given size; assigning, by the process, the file to a particular size-range bucket of a plurality of size-range buckets of a data ingestion pipeline, the plurality of size-range buckets having a corresponding size-based configuration; and forwarding, by the process, the file into a particular size-based service mesh ingress gateway of the data ingestion pipeline according to the particular size-range bucket to cause processing of the file within the data ingestion pipeline according to the corresponding size-based configuration of the particular size-range bucket.

Various embodiments relate generally to data science and data analysis, and computer software and systems to provide an interface between repositories of disparate datasets and computing machine-based entities that seek access to the datasets, and, more specifically, to a computing and data storage platform that facilitates consolidation of one or more datasets, whereby data ingestion is performed to form data representing layered data files and data arrangements to facilitate, for example, interrelations among a system of networked collaborative datasets. In some examples, a method may include forming a first layer data file and a second layer data file, assigning addressable identifiers to uniquely identify units of data and data units to facilitate the linking of data, and implementing selectively one or more of a unit of data and a data unit as a function of a context of a data access request for a collaborative dataset.

The application discloses a storage method applied to an access device, and the method comprises the following steps: in the case that a storage request for a first file is received, an identification code is allocated to the first file; an index number of the first file is generated by using an existing parent directory of the first file and the identification code; an association between a name of the first file and the index number of the first file is stored; and the first file and the index number of the first file are sent to an index device. The application also discloses a device, a system and a computer storage medium, and the reliability of file storage is improved, and the storage efficiency of file storage is improved.

The invention provides a data analysis method. The data analysis method comprises the following steps that a list of files to be analyzed is obtained according to a file path, and a file queue is formed; distributing the files in the file queue to N threads determined by the number of processor cores to realize load balancing; searching packet headers and packet tails of the data packets from the distributed file data in parallel through N threads so as to identify and extract complete or partial data packets; and carrying out analysis and integrity verification on the extracted data packet, and writing the analyzed data into a database system. According to the data analysis method provided by the invention, through multi-thread parallel processing and load balanced distribution, the data analysis efficiency is remarkably improved, and the overall processing time is shortened.

The invention provides a file analysis system which comprises a file reading module used for obtaining the name and the size of a to-be-analyzed file according to a file path and putting the file into a file queue; the file distribution module is used for sorting the files in the file queue according to the sizes of the files, creating a plurality of data buffer areas and a plurality of threads according to the core number of the processor, and distributing the files to the threads by adopting a load balancing method; each analysis execution module runs in an independent thread, and comprises a file search unit used for identifying and extracting a complete or partial data packet from file data distributed to the thread by searching a packet header and a packet tail of the data packet; and the file analysis unit is used for carrying out analysis and integrity verification on the data packet extracted by the file search unit and writing the analyzed data into a database system. According to the system provided by the invention, multi-file parallel analysis can be realized, and the analysis efficiency and the system throughput are remarkably improved.

File system full and incremental replication to object storage systems (e.g., using a computerized tool), is enabled. For example, a system can comprise at least one processor, and at least one memory that stores executable instructions that, when executed by the at least one processor, facilitate performance of operations. The operations can comprise identifying a file system, wherein the file system comprises a group of files, based on an attribute applicable to the file system, determining a dataset identifier representative of the file system, for respective files represented in the group of files, allocating the dataset identifier and allocating respective file identifiers, and copying the group of files to an object store with respective object names corresponding to the respective files, wherein the copying comprises, for the respective files, assigning the dataset identifier to respective object names and assigning the respective file identifiers to the respective object names.

The invention belongs to the technical field of computer storage, and particularly relates to a ZNS SSD partition allocation optimization method based on machine learning. Comprising the following steps: constructing a file life prediction system based on a machine learning model, and establishing a file life prediction model according to an overlapping rate between a hierarchy where an SST file is created and a next hierarchy file, a total file size of the hierarchy, a global compression operation count, an interval between the hierarchy and a previous compression operation, and a latest global compression operation count of the hierarchy. The file life range is accurately predicted as features; implementing a partition allocation algorithm based on life labels, and preferentially allocating files with the same life labels to the same partition according to the predicted file life labels; and designing a time-aware dynamic partition allocation strategy, measuring the partition life according to the predicted life label, and dynamically adjusting the partition life label according to time lapse. According to the method, the data migration amount during garbage collection is remarkably reduced, write amplification is reduced, the throughput of the ZNS SSD is improved, and the service life of the ZNS SSD is prolonged.

The application provides a method and system for stream storage management and persistence of files, and relates to the technical field of computer storage, which comprises the following steps: dividing a hard disk storage space into a plurality of file blocks with the same size to obtain a plurality of initial state unused file blocks; creating and managing a plurality of virtual domains based on the obtained plurality of initial state unused file blocks; allocating the plurality of initial state unused file blocks to different virtual domains for management; when a to-be-written file is received according to the established virtual domain, matching to the corresponding virtual domain based on the attribute of the to-be-written file, and allocating a target file block and a unique file serial number for the to-be-written file in the corresponding virtual domain; and generating a corresponding file ID based on the allocated target file block and unique file serial number. The application realizes low-latency, high-throughput and sequential read-write storage management of sequentially, massively and quickly continuously arriving stream files in application scenarios.

The invention provides an SATA (Serial Advanced Technology Attachment) disk array access method, which comprises the following steps of: acquiring write-in data of FIFO (First In First Out), carrying out circular detection on the data volume written in the FIFO based on a timestamp, initiating a file write-in application, and carrying out file distribution on the written-in data volume so as to correspondingly write the write-in data into an SATA write queue, the file distribution is based on disk information of each SATA disk, and the disk information at least comprises disk capacity and a relative wear factor; arbitrating each path of SATA write queue through an arbitration strategy, performing data caching on the arbitrated SATA write queue, and sending a write completion message; and reading the cached data according to the received write completion message, forming a cache queue, and transmitting the data in the cache queue to a rear-end SATA disk. According to the method, a multi-level storage strategy is adopted, so that the high-speed reliability and anti-risk capability of the multi-level cache can be improved, and the power consumption cost of the system is greatly saved.

This application provides a distributed data reading method, apparatus, and server, relating to the field of data processing. The method includes: a server retrieving multiple file sets from a disk. Each file set may include at least one file. Each file includes metadata for at least one document. The server determines the number of files allocated in each partition based on the quotient of the total number of files in each file set and the number of parallel processes. The server may allocate the files in the file sets to several partitions of parallel processes. Each partition includes at least several allocated files. After determining the number of partitions for the parallel processes, the server may allocate one partition to each parallel process. The parallel process may then read the document corresponding to the metadata in the partition after retrieving it. This method improves data reading efficiency.

Provided are a device, system, method, and computer program for inferring an attacker group by analyzing malicious code. The system includes a sandbox pool manager configured to allocate analysis target files for inferring an attacker group to one or more nodes and separately execute the analysis target files in separate malicious code analysis environments by controlling each node, an event manager configured to determine in real time whether all events related to the analysis target files have been collected on the basis of running state information of each node and collect events which are recorded in the malicious code analysis environments of each of the nodes and related to the analysis target files, an attacker group inference part configured to infer an attacker group by analyzing the collected events, and an analysis result provider configured to provide information on the inferred attacker group.

The application provides a file system processing method, system, server, storage medium and program product. The method is suitable for a container migration scene. In a container saving stage, a source end reads original device numbers of mounting points of original tmpfs and original inode numbers of files in a source container, and transmits the original device numbers and the original inode numbers to a target end through a container management end. In a container recovery stage, the target end explicitly allocates corresponding original device numbers to created mounting points when the mounting points are created, so that consistency of the device numbers of the tmpfs before and after saving and recovery can be ensured. When files are created in the mounting points, corresponding original inode numbers are explicitly allocated to the created files, so that consistency of inode numbers of the files before and after saving and recovery can be ensured, abnormality / failure caused by inconsistency of the device numbers and the inode numbers of the tmpfs before and after migration can be avoided, and safety and stability of a system are improved.

The application relates to the technical field of file analysis, and discloses a user package file analysis method, system, device and medium based on load balancing, which comprises the following steps: carrying out file splitting on a user package file to obtain a user file list; selecting a target server cluster in a preset backend server cluster according to the user file list; calculating the real-time computing resources of each backend server in the target server cluster, and constructing a file allocation strategy according to the real-time computing resources; carrying out file analysis according to the file allocation strategy, calculating the analysis real-time computing resources after file analysis; and iteratively optimizing the file allocation strategy according to the analysis real-time computing resources until each user file is analyzed to obtain the analysis result of the user package file. According to the application, the user files can be reasonably allocated for file analysis according to the real-time load conditions of the backend servers, so that the computing resources of the backend server cluster can be reasonably utilized, and the stability and accuracy of user file analysis can be effectively improved.

The embodiment of the invention provides a file distribution method and device, a storage medium and a program product, and relates to the field of distribution. The method comprises the steps of obtaining a to-be-stored file; determining an allocation unit of the to-be-stored file according to the file type of the to-be-stored file; wherein the file type is divided based on the size of the to-be-stored file, and when the size of the to-be-stored file is smaller than a preset threshold value, the file type is a small file, and the block is selected as a distribution unit of the to-be-stored file; when the size of the to-be-stored file is greater than or equal to a preset threshold value and the file type is a large file, selecting an interval as a distribution unit of the to-be-stored file; and distributing a target storage area for the to-be-stored file in the storage space according to the distribution unit of the to-be-stored file. According to the method, the distribution flexibility of file storage is improved, and the overall access performance and the system response speed of different types of files are improved.

The invention discloses a full life cycle tracking management method for engineering surveying and mapping archives, relates to the technical field of engineering surveying and mapping archives, and aims to solve the technical problem that low-frequency access historical archives occupy high-cost hot storage resources for a long time. Comprising the following steps: S1, digital identity creation: allocating a dynamic unique identification code DUI to an achievement file generated by engineering surveying and mapping field data processing by adopting a one-way Hash binding algorithm, and generating an initial Hash value after binding metadata with the DUI and writing the initial Hash value into a block chain; and S2, auditing and warehousing: the archives passing the quality auditing are marked as in-library states, are stored in a high-performance online storage library as hot data, and are chained for evidence storage. According to the method, the low-frequency access archives are automatically identified through the quantitative judgment logic of the preset access frequency and the time interval threshold value, the problems that low-frequency archives occupy resources and high-frequency archives are high in delay in traditional centralized storage are effectively solved, and the problem that low-frequency access historical archives occupy high-cost hot storage resources for a long time is solved.

The present invention relates to an information processing apparatus, a control method thereof, and a storage medium. A user is enabled to more easily set rules related to folder assignment and file naming when filing and storing files such as document scan images and received faxes. On a user interface screen for setting rules related to attribute information used when assigning files to folders for storage, at least a first area for editing the rules and a second area for displaying items selectable by the user are provided. The items are unit elements for the user to set strings configuring attribute information, and correspond to strings of arbitrary attributes settable by the user. Then, the rules are created by unit items selected by the user from the second area, which is arranged at a position specified by the user in the first area.

The invention discloses a file allocation method based on a file storage cluster and a related device, and the file allocation method comprises the steps: responding to a received allocation request about a current to-be-allocated file, aiming at each storage device, according to the stability degree of the storage capacity of each allocated file stored in the storage device, allocating the current to-be-allocated file according to the stability degree of the storage capacity of each allocated file stored in the storage device; dividing each distributed file stored in the storage device into a static file and a dynamic file; counting the current storage capacity of each static file stored in the storage device to obtain a static storage capacity, and counting the future storage capacity of each dynamic file stored in the storage device to obtain a dynamic storage capacity; subtracting the static storage capacity and the dynamic storage capacity from the total storage capacity of the storage device to obtain the available storage capacity of the storage device; and distributing the current to-be-distributed file to a storage device of which the available storage capacity meets the storage capacity condition for storage. According to the scheme, the balance degree of storage space utilization rate distribution of the storage devices can be improved.

The invention discloses a heterogeneous text offline transmission method and system based on dual-metadata collaboration and adaptive FEC (Forward Error Correction), and the method comprises the steps: distributing a file identifier to a to-be-transmitted text file, partitioning the to-be-transmitted text file, and setting an overlapping region to obtain a data block with the overlapping region; and generating a dual-metadata file containing position and content metadata files. Selecting a forward error correction coding mode for coding according to a transmission environment, and generating a two-dimensional code after converting a text character string in a uniform format and adding a check code; and decoding and identifying the two-dimensional code, storing the data in the dual-metadata pool in a classified manner after verification, extracting the data, decoding, recovering the metadata file and the data block with the overlapping region, and verifying. And finally, sequencing the verified data blocks, verifying the continuity of the overlapping regions, removing overlapping region splicing, restoring the file structure according to the position metadata file, and completing transmission and restoration. The technical problems that in the prior art, in the off-line transmission process of a large number of texts, the structure is lost, the error correction capacity is weak, and recovery is difficult are solved.

The invention provides a file resource pool allocation method and device, equipment, a storage medium and a program product, relates to the field of big data, and can be applied to the field of cloud application. The method comprises the following steps: in response to a file creation request of a file system, modifying a kernel driver of the file system so as to intercept the file creation request when an index node is allocated to a file corresponding to the file creation request; acquiring a global storage strategy table; wherein the global storage strategy table comprises a plurality of strategies, and the strategies comprise file name wildcard characters and identifiers of corresponding target storage pools; extracting a file name of the file; determining an identifier of a target storage pool corresponding to the file according to the file name and a global storage strategy table; writing an identifier of a target storage pool corresponding to the file into the index node; wherein the index node is used for indicating to distribute the file to the target storage pool. The flexibility of file resource allocation is improved, the manual scheduling cost is reduced, and the overall performance of a system is improved.

The application discloses a kind of based on the method for the cooperative work of multiple programmable controllers of data synchronization, comprising the following steps: S1, variable definition, export variable declaration and program writing;S2, parse program file XML, allocate the variable address of each programmable controller variable, process each pair of export variable relationship;S3, program compilation, the synchronization function block provided by the application is essentially a function function encapsulating synchronization code, but it is transparent to the user, the user does not need to worry about its implementation problem, the implementation of code will be generated according to the difference of export variable and the difference of programmable controller device information;The application uses communication protocol, and the export variable of the same programmable controller is synchronized once, which well solves the problem of too many receiving and transmitting frames caused by synchronizing only a single variable each time, finally makes the waiting time too long.

Embodiments of the present invention provide a resource allocation method, a resource processing method, a device, a storage medium, and a program product. In the resource allocation method, a resource allocation request of a Pod is acquired, and then a target resource allocation component is called to pre-allocate to the Pod a plurality of groups of huge-page memory files corresponding to a plurality of NUMA nodes on a target node, and returns directories of the plurality of groups of huge-page memory files to a node agent. The node agent allocates the plurality of groups of huge-page memory files to the Pod. Therefore, when a container orchestration engine does not support allocation, based on the granularity of NUMA nodes, of huge-page memory resources to the Pod, pre-allocation of the huge-page memory resources based on the granularity of the NUMA nodes is achieved by means of the target resource allocation component, thereby facilitating containerized deployment of a target application in the Pod and satisfying the high-performance processing requirements of the target application. In addition, the target application in the Pod does not need to create huge-page memory files during startup, thereby further increasing the startup speed of the target application.

Various embodiments relate generally to data science and data analysis, and computer software and systems to provide an interface between repositories of disparate datasets and computing machine-based entities that seek access to the datasets, and, more specifically, to a computing and data storage platform that facilitates consolidation of one or more datasets, whereby data ingestion is performed to form data representing layered data files and data arrangements to facilitate, for example, interrelations among a system of networked collaborative datasets. In some examples, a method may include forming a first layer data file and a second layer data file, assigning addressable identifiers to uniquely identify units of data and data units to facilitate the linking of data, and implementing selectively one or more of a unit of data and a data unit as a function of a context of a data access request for a collaborative dataset.

Systems, apparatuses, methods, and computer program products are disclosed for linking remote file storage. An example method includes identifying, by discovery agent circuitry, a set of files for storage from a set of candidate files, and classifying, by file metadata circuitry, each file from the set of files for storage into file usage categories. The example method also includes assigning, by file metadata circuitry, permission categories to each file from the set of files for storage based on the file usage categories, and providing, by communications hardware, the set of files for storage and associated metadata to a file storage host, wherein the associated metadata comprises the file usage categories and the permission categories The example method also includes updating, by the discovery agent circuitry, content stored on the file storage host, the content comprising the set of files for storage.

The application provides a stream management method and a computing device for writing data to a multi-stream solid state disk. The stream management method is based on a user state and includes: receiving a write file request for writing a file; distributing the write file to at least one virtual stream according to an adaptive distribution strategy, each virtual stream including at least one write data block request of a data block, and the at least one write data block request of the data block being assigned with a same virtual stream ID; calculating the heat of each virtual stream; and mapping each virtual stream to a physical stream based on the calculated heat. According to the technical solution of the embodiments, the performance of the multi-stream solid state disk can be improved through the adaptive distribution strategy and the virtual stream mapping.

The invention provides an analysis report generation method and system based on vehicle model comment data, and belongs to the technical field of vehicle data analysis. The method comprises the following steps: constructing a file-level thread pool, distributing to-be-processed vehicle model comment files to different threads of the thread pool, calling a large model in parallel based on the different threads, and performing multi-dimensional comment feature extraction on comment data in the vehicle model comment files by utilizing the large model to obtain initial tags; judging whether to call a large model to obtain a new label or not by combining confidence, and generating an analysis report based on the initial label and the new label. According to the embodiment of the invention, the large model is called in parallel through multiple threads, the multi-source data from the comment files of different vehicle types are effectively integrated, and the accuracy and processing efficiency of the analysis report are remarkably improved through multi-dimensional feature extraction, automatic comment data annotation and generation of the analysis report.

The application provides a kind of electronic archive file fast checking method, and the compression file package is sorted in the storage phase according to the name string rule order sibling file, and the corresponding thread resource is distributed to start from the lowermost layer of folder and gradually calculate the file HASH value upwards, build layer HASH file and associate relevant file saved in current layer, build HASH directory tree, and pack to generate archive electronic archive package;In the verification stage, arrange the file and splice string according to the arrangement mode of the verification stage, calculate the HASH value of each layer of subfolder from top to bottom, compare with the corresponding level HASH file saved, generate the level and position information of abnormal file, and locate the change file and change position of electronic archive.The application realizes the traceability of electronic archive storage, and guarantees the authenticity of storage electronic archive.