43 results about "File allocation" patented technology

A storage system includes a storage device and a host device. The storage device includes a non-volatile memory device configured to store data and a storage controller configured to control the non-volatile memory device to isolate and store the data by grouping storage spaces of the non-volatile memory device into a plurality of isolated storage spaces. The host device includes a file system configured to generate and manage a plurality of file allocation instances corresponding to a plurality of file allocation system calls from an application program by allocating, with respect to each of the plurality of file allocation system calls, a logical address range and an isolated identifier of a plurality of isolated identifiers corresponding to the plurality of isolated storage spaces.

Systems, apparatuses, methods, and computer program products are disclosed for linking remote file storage. An example method includes identifying, by discovery agent circuitry, a set of files for storage from a set of candidate files, and classifying, by file metadata circuitry, each file from the set of files for storage into file usage categories. The example method also includes assigning, by file metadata circuitry, permission categories to each file from the set of files for storage based on the file usage categories, and providing, by communications hardware, the set of files for storage and associated metadata to a file storage host, wherein the associated metadata comprises the file usage categories and the permission categories The example method also includes updating, by the discovery agent circuitry, content stored on the file storage host, the content comprising the set of files for storage.

The invention discloses a file processing method and device, equipment, a storage medium and a computer program product, and the method comprises the steps: obtaining the attribute information of a file in response to a file processing request, and determining the to-be-read data volume of the file; performing priority ranking on the files based on the attribute information and the to-be-read data volume to obtain a ranking result; distributing the file to a first queue or a second queue according to the sorting result, the attribute information and the current processing capability; wherein the first queue is configured to instantly process the file, and the second queue is configured to delay processing of the file. Through the design of a flow pool system and double-queue parallel processing, the system load can be effectively balanced, resource waste is avoided, and quick response of file processing is ensured. Besides, the automatic length adjustment of the processing queue enables the system to better adapt to the fluctuation of the business volume, and the system can keep stable operation, reduce the failure rate and improve the user experience no matter whether the peak or the valley is processed.

In one embodiment, an illustrative method herein comprises: obtaining, by a process, a file having a given size; assigning, by the process, the file to a particular size-range bucket of a plurality of size-range buckets of a data ingestion pipeline, the plurality of size-range buckets having a corresponding size-based configuration; and forwarding, by the process, the file into a particular size-based service mesh ingress gateway of the data ingestion pipeline according to the particular size-range bucket to cause processing of the file within the data ingestion pipeline according to the corresponding size-based configuration of the particular size-range bucket.

A dynamic data extraction, distillation, and enhancement system is disclosed that includes a dynamic extraction, distillation, and enhancement framework. The framework includes an allocator, extractor, and deconstructor stored in a non-transitory memory that, when executed by a processor, receive files in different formats from data sources, determine a native format of each file, identify and extract an embedded object from a file, deconstruct the file into components, assign each file to one of a plurality of streams based on the native format of the file, assign the embedded object to a stream based on a format of the embedded object, and assign a deconstructed component to a stream based on a format of the deconstructed component. The native format includes one of text, video, image, or audio. Each stream corresponds to one native format. The streams include a text stream, an audio stream, a video stream, and an image stream.

This invention discloses a file upload method, apparatus, device, and medium. The method includes: obtaining network bandwidth rate, storage space of a target file, and the number of available data nodes; calculating the storage capacity of each available data node based on the storage space of the target file and the number of available data nodes; splitting the target file according to the network bandwidth rate to obtain at least one target sub-file; the target file includes at least one target sub-file; allocating each target sub-file to each available data node according to the storage capacity of each available data node, obtaining a mapping relationship between each target sub-file and each available data node; and uploading each target sub-file to each available data node according to the mapping relationship. Embodiments of this invention can improve the efficiency of file transfer.

Various embodiments relate generally to data science and data analysis, and computer software and systems to provide an interface between repositories of disparate datasets and computing machine-based entities that seek access to the datasets, and, more specifically, to a computing and data storage platform that facilitates consolidation of one or more datasets, whereby data ingestion is performed to form data representing layered data files and data arrangements to facilitate, for example, interrelations among a system of networked collaborative datasets. In some examples, a method may include forming a first layer data file and a second layer data file, assigning addressable identifiers to uniquely identify units of data and data units to facilitate the linking of data, and implementing selectively one or more of a unit of data and a data unit as a function of a context of a data access request for a collaborative dataset.

To enable a user to more easily set a rule relating to folder allocation and file naming at the time of filing data, such as a document scanned image and a received facsimile, and storing the file. On a user interface screen for setting a rule relating to property information that is used at the time of allocating a file to a folder for storage, at least a first area for editing the rule and a second area displaying an item a user can select are provided. The item is a unit element for a user to set a character string configuring the property information and corresponds to a character string of an arbitrary attribute a user can set. Then, the rule is created by the item selected by a user from the second area being arranged at a position designated by the user in the first area.

The application discloses a storage method applied to an access device, and the method comprises the following steps: in the case that a storage request for a first file is received, an identification code is allocated to the first file; an index number of the first file is generated by using an existing parent directory of the first file and the identification code; an association between a name of the first file and the index number of the first file is stored; and the first file and the index number of the first file are sent to an index device. The application also discloses a device, a system and a computer storage medium, and the reliability of file storage is improved, and the storage efficiency of file storage is improved.

The invention provides a data analysis method. The data analysis method comprises the following steps that a list of files to be analyzed is obtained according to a file path, and a file queue is formed; distributing the files in the file queue to N threads determined by the number of processor cores to realize load balancing; searching packet headers and packet tails of the data packets from the distributed file data in parallel through N threads so as to identify and extract complete or partial data packets; and carrying out analysis and integrity verification on the extracted data packet, and writing the analyzed data into a database system. According to the data analysis method provided by the invention, through multi-thread parallel processing and load balanced distribution, the data analysis efficiency is remarkably improved, and the overall processing time is shortened.

This invention discloses a storage system, read / write method, and controller for reducing file fragmentation, belonging to the field of data storage technology. The system includes a host and a solid-state drive (SSD). The host has a file system and a space pre-allocator, while the SSD has a segment allocator. The file system allocates logical address space to files according to segments. The segment allocator maintains a global translation table, which records the mapping relationship between segment IDs and mapped page addresses. The mapped pages are flash pages used to record the logical page addresses within a segment to the physical page addresses. The segment allocator also calculates free segment IDs in a monotonically increasing manner and publishes them to the circular buffer when there is remaining space. The space pre-allocator retrieves the free segment ID from the head of the circular buffer and allocates it to the file when space needs to be allocated. This invention can reduce the generation of file fragmentation and free space fragmentation in the file system, improving the overall performance and stability of the system.

The invention provides a file analysis system which comprises a file reading module used for obtaining the name and the size of a to-be-analyzed file according to a file path and putting the file into a file queue; the file distribution module is used for sorting the files in the file queue according to the sizes of the files, creating a plurality of data buffer areas and a plurality of threads according to the core number of the processor, and distributing the files to the threads by adopting a load balancing method; each analysis execution module runs in an independent thread, and comprises a file search unit used for identifying and extracting a complete or partial data packet from file data distributed to the thread by searching a packet header and a packet tail of the data packet; and the file analysis unit is used for carrying out analysis and integrity verification on the data packet extracted by the file search unit and writing the analyzed data into a database system. According to the system provided by the invention, multi-file parallel analysis can be realized, and the analysis efficiency and the system throughput are remarkably improved.

File system full and incremental replication to object storage systems (e.g., using a computerized tool), is enabled. For example, a system can comprise at least one processor, and at least one memory that stores executable instructions that, when executed by the at least one processor, facilitate performance of operations. The operations can comprise identifying a file system, wherein the file system comprises a group of files, based on an attribute applicable to the file system, determining a dataset identifier representative of the file system, for respective files represented in the group of files, allocating the dataset identifier and allocating respective file identifiers, and copying the group of files to an object store with respective object names corresponding to the respective files, wherein the copying comprises, for the respective files, assigning the dataset identifier to respective object names and assigning the respective file identifiers to the respective object names.

The invention provides a file system processing method and system, a server, a storage medium and a program product. The method is suitable for a container migration scene, in a container storage stage, a source end reads an original device number of a mounting point of original tmpfs in a source container and an original index node number of a file, and transmits the original device number and the original index node number to a target end through a container management and control end; in the container recovery stage, the target end explicitly allocates the corresponding original device number to the created mounting point when creating the mounting point, so that the consistency of the device numbers before and after the tmpfs is stored and recovered can be ensured. When the file is created in the mounting point, the corresponding original index node number is explicitly allocated to the created file, so that the consistency of the file index node numbers before and after the tmpfs is stored and recovered can be ensured, the abnormality / fault caused by the inconsistency of the device number and the index node number before and after the tmpfs is migrated can be avoided, and the security and stability of the system are improved.

The invention belongs to the technical field of computer storage, and particularly relates to a ZNS SSD partition allocation optimization method based on machine learning. Comprising the following steps: constructing a file life prediction system based on a machine learning model, and establishing a file life prediction model according to an overlapping rate between a hierarchy where an SST file is created and a next hierarchy file, a total file size of the hierarchy, a global compression operation count, an interval between the hierarchy and a previous compression operation, and a latest global compression operation count of the hierarchy. The file life range is accurately predicted as features; implementing a partition allocation algorithm based on life labels, and preferentially allocating files with the same life labels to the same partition according to the predicted file life labels; and designing a time-aware dynamic partition allocation strategy, measuring the partition life according to the predicted life label, and dynamically adjusting the partition life label according to time lapse. According to the method, the data migration amount during garbage collection is remarkably reduced, write amplification is reduced, the throughput of the ZNS SSD is improved, and the service life of the ZNS SSD is prolonged.

The invention discloses a file processing method, a file system and a readable storage medium. The method is applied to a system comprising a private client, a master file server and fragmented file servers, the master file server segments a target file and distributes obtained N fragmented files to M file servers for management, and the method comprises the following steps: the private client performs data distribution on the fragmented files according to a first data layout table, when it is determined that a first fragment identifier of to-be-written first data is larger than a first maximum fragment identifier, first fragment creation information is sent to the master file server, so that the master file server creates a first fragment, the maximum fragment identifier included in the second data layout table is updated to be the first fragment identifier, and the first fragment is written into the second data layout table; and sending the updated second data layout table to the first private client. According to the fragment creation process, the accuracy of the data layout table can be ensured by updating the data layout table, so that the accuracy of file processing operations such as attribute acquisition executed by a subsequent private client is ensured.

A system and method for managing unstructured data stored across multiple data repositories is disclosed. The method includes accessing unstructured data items from distinct data repositories and generating classification metadata for each unstructured data item using a machine learning-based classifier. A residency pointer is generated for each unstructured data item based on residency metadata that identifies its storage location within the data repositories. An unstructured data metafile is automatically constructed for each unstructured data item. Each unstructured data metafile is assigned to one or more virtual queues based on its classification metadata, enabling organized data management. The method further integrates the virtual queues with a web-based data handling interface that facilitates batch operations on unstructured data items stored across the distinct repositories. The disclosed approach improves data classification, retrieval efficiency, and governance of unstructured data without requiring physical relocation of stored data items.

The application provides a method and system for stream storage management and persistence of files, and relates to the technical field of computer storage, which comprises the following steps: dividing a hard disk storage space into a plurality of file blocks with the same size to obtain a plurality of initial state unused file blocks; creating and managing a plurality of virtual domains based on the obtained plurality of initial state unused file blocks; allocating the plurality of initial state unused file blocks to different virtual domains for management; when a to-be-written file is received according to the established virtual domain, matching to the corresponding virtual domain based on the attribute of the to-be-written file, and allocating a target file block and a unique file serial number for the to-be-written file in the corresponding virtual domain; and generating a corresponding file ID based on the allocated target file block and unique file serial number. The application realizes low-latency, high-throughput and sequential read-write storage management of sequentially, massively and quickly continuously arriving stream files in application scenarios.

The invention provides an SATA (Serial Advanced Technology Attachment) disk array access method, which comprises the following steps of: acquiring write-in data of FIFO (First In First Out), carrying out circular detection on the data volume written in the FIFO based on a timestamp, initiating a file write-in application, and carrying out file distribution on the written-in data volume so as to correspondingly write the write-in data into an SATA write queue, the file distribution is based on disk information of each SATA disk, and the disk information at least comprises disk capacity and a relative wear factor; arbitrating each path of SATA write queue through an arbitration strategy, performing data caching on the arbitrated SATA write queue, and sending a write completion message; and reading the cached data according to the received write completion message, forming a cache queue, and transmitting the data in the cache queue to a rear-end SATA disk. According to the method, a multi-level storage strategy is adopted, so that the high-speed reliability and anti-risk capability of the multi-level cache can be improved, and the power consumption cost of the system is greatly saved.

This application provides a distributed data reading method, apparatus, and server, relating to the field of data processing. The method includes: a server retrieving multiple file sets from a disk. Each file set may include at least one file. Each file includes metadata for at least one document. The server determines the number of files allocated in each partition based on the quotient of the total number of files in each file set and the number of parallel processes. The server may allocate the files in the file sets to several partitions of parallel processes. Each partition includes at least several allocated files. After determining the number of partitions for the parallel processes, the server may allocate one partition to each parallel process. The parallel process may then read the document corresponding to the metadata in the partition after retrieving it. This method improves data reading efficiency.

Provided are a device, system, method, and computer program for inferring an attacker group by analyzing malicious code. The system includes a sandbox pool manager configured to allocate analysis target files for inferring an attacker group to one or more nodes and separately execute the analysis target files in separate malicious code analysis environments by controlling each node, an event manager configured to determine in real time whether all events related to the analysis target files have been collected on the basis of running state information of each node and collect events which are recorded in the malicious code analysis environments of each of the nodes and related to the analysis target files, an attacker group inference part configured to infer an attacker group by analyzing the collected events, and an analysis result provider configured to provide information on the inferred attacker group.

The invention discloses a bridge data sub-library and model scheduling method, device and equipment based on information grading protection, and relates to the field of bridge industry knowledge management and intelligent question answering, and the method comprises the steps: according to the confidentiality requirements of different levels, based on a bridge design, construction and operation and maintenance file confidentiality level intelligent discrimination algorithm, distinguishing the confidentiality levels; in combination with an active auditing mode, the uploaded bridge design construction operation and maintenance files are distributed to bridge knowledge databases with different confidentiality levels; and obtaining input related questioning contents of the bridge industry, judging secret-related conditions of the related questioning contents of the bridge industry based on the bridge design, construction and operation and maintenance file security classification intelligent judgment algorithm, and scheduling a corresponding large language model and a bridge knowledge database to organize answers according to the current authority level. According to the invention, the safety and intelligent level of bridge industry knowledge management can be improved.

The application provides a file system processing method, system, server, storage medium and program product. The method is suitable for a container migration scene. In a container saving stage, a source end reads original device numbers of mounting points of original tmpfs and original inode numbers of files in a source container, and transmits the original device numbers and the original inode numbers to a target end through a container management end. In a container recovery stage, the target end explicitly allocates corresponding original device numbers to created mounting points when the mounting points are created, so that consistency of the device numbers of the tmpfs before and after saving and recovery can be ensured. When files are created in the mounting points, corresponding original inode numbers are explicitly allocated to the created files, so that consistency of inode numbers of the files before and after saving and recovery can be ensured, abnormality / failure caused by inconsistency of the device numbers and the inode numbers of the tmpfs before and after migration can be avoided, and safety and stability of a system are improved.

The application relates to the technical field of file analysis, and discloses a user package file analysis method, system, device and medium based on load balancing, which comprises the following steps: carrying out file splitting on a user package file to obtain a user file list; selecting a target server cluster in a preset backend server cluster according to the user file list; calculating the real-time computing resources of each backend server in the target server cluster, and constructing a file allocation strategy according to the real-time computing resources; carrying out file analysis according to the file allocation strategy, calculating the analysis real-time computing resources after file analysis; and iteratively optimizing the file allocation strategy according to the analysis real-time computing resources until each user file is analyzed to obtain the analysis result of the user package file. According to the application, the user files can be reasonably allocated for file analysis according to the real-time load conditions of the backend servers, so that the computing resources of the backend server cluster can be reasonably utilized, and the stability and accuracy of user file analysis can be effectively improved.

To solve such a problem that the stand-by time of a user is long if a series of processing to reception of a confirmation result of the user through a UI from recommendation is simply repeated when recommending a file name or the like by scanning a plurality of documents at once and dividing the documents to different files.SOLUTION: An image processing device acquires image data of a plurality of documents, divides the image data to create a plurality of files, executes OCR-associated processing, performs matching between the document of the file and a registered document and determines the type of the document. The image processing device generates a recommendation content about an index from the document of the file and presents it to a user in accordance with an index extraction rule linked to the registered document, executes correction processing of the index and the index extraction rule on the basis of a correction instruction of the user, and gives the index to the file. The image processing device performs parallel processing of the first processing including the OCR-associated processing and the second processing including the correction processing.SELECTED DRAWING: Figure 3

The embodiment of the invention provides a file distribution method and device, a storage medium and a program product, and relates to the field of distribution. The method comprises the steps of obtaining a to-be-stored file; determining an allocation unit of the to-be-stored file according to the file type of the to-be-stored file; wherein the file type is divided based on the size of the to-be-stored file, and when the size of the to-be-stored file is smaller than a preset threshold value, the file type is a small file, and the block is selected as a distribution unit of the to-be-stored file; when the size of the to-be-stored file is greater than or equal to a preset threshold value and the file type is a large file, selecting an interval as a distribution unit of the to-be-stored file; and distributing a target storage area for the to-be-stored file in the storage space according to the distribution unit of the to-be-stored file. According to the method, the distribution flexibility of file storage is improved, and the overall access performance and the system response speed of different types of files are improved.

The invention discloses a full life cycle tracking management method for engineering surveying and mapping archives, relates to the technical field of engineering surveying and mapping archives, and aims to solve the technical problem that low-frequency access historical archives occupy high-cost hot storage resources for a long time. Comprising the following steps: S1, digital identity creation: allocating a dynamic unique identification code DUI to an achievement file generated by engineering surveying and mapping field data processing by adopting a one-way Hash binding algorithm, and generating an initial Hash value after binding metadata with the DUI and writing the initial Hash value into a block chain; and S2, auditing and warehousing: the archives passing the quality auditing are marked as in-library states, are stored in a high-performance online storage library as hot data, and are chained for evidence storage. According to the method, the low-frequency access archives are automatically identified through the quantitative judgment logic of the preset access frequency and the time interval threshold value, the problems that low-frequency archives occupy resources and high-frequency archives are high in delay in traditional centralized storage are effectively solved, and the problem that low-frequency access historical archives occupy high-cost hot storage resources for a long time is solved.

The present invention relates to an information processing apparatus, a control method thereof, and a storage medium. A user is enabled to more easily set rules related to folder assignment and file naming when filing and storing files such as document scan images and received faxes. On a user interface screen for setting rules related to attribute information used when assigning files to folders for storage, at least a first area for editing the rules and a second area for displaying items selectable by the user are provided. The items are unit elements for the user to set strings configuring attribute information, and correspond to strings of arbitrary attributes settable by the user. Then, the rules are created by unit items selected by the user from the second area, which is arranged at a position specified by the user in the first area.

The invention discloses a file allocation method based on a file storage cluster and a related device, and the file allocation method comprises the steps: responding to a received allocation request about a current to-be-allocated file, aiming at each storage device, according to the stability degree of the storage capacity of each allocated file stored in the storage device, allocating the current to-be-allocated file according to the stability degree of the storage capacity of each allocated file stored in the storage device; dividing each distributed file stored in the storage device into a static file and a dynamic file; counting the current storage capacity of each static file stored in the storage device to obtain a static storage capacity, and counting the future storage capacity of each dynamic file stored in the storage device to obtain a dynamic storage capacity; subtracting the static storage capacity and the dynamic storage capacity from the total storage capacity of the storage device to obtain the available storage capacity of the storage device; and distributing the current to-be-distributed file to a storage device of which the available storage capacity meets the storage capacity condition for storage. According to the scheme, the balance degree of storage space utilization rate distribution of the storage devices can be improved.