3373 results about "File system" patented technology

A video streaming system including a video cache system having a cache database and a video storage device, and including a video server configured to communicate with a gateway system configured to store multiple video streams, communicate with a cloud backup system configured to store video in a file system, and communicate with a cloud server to receive a stream request, the stream request indicating requested video that comprises multiple video segments. The video server is further configured to receive the stream request and determine a playlist that indicates where on the video cache system, the cloud backup system, and the gateway system the video segments that are needed to fulfill the stream request are stored, and to communicate needed video segments to the video cache system in preparation for providing streaming video to fulfill the stream request.

An anomaly detection system is disclosed capable of reporting anomalous processes or hosts in a computer network using machine learning models trained using unsupervised training techniques. In embodiments, the system assigns observed processes to a set of process categories based on the file system path of the program executed by the process. The system extracts a feature vector for each process or host from the observation records and applies the machine learning models to the feature vectors to determine an outlier metric each process or host. The processes or hosts with the highest outlier metrics are reported as detected anomalies to be further examined by security analysts. In embodiments, the machine learnings models may be periodically retrained based on new observation records using unsupervised machine learning techniques. Accordingly, the system allows the models to learn from newly observed data without requiring the new data to be manually labeled by humans.

The invention relates to the technical field of data storage and processing, in particular to an RAID card static cache management method and device based on data popularity, and the method comprises the steps that the physical position of a data block needing to be accessed in an SSD array is acquired according to a read-write request; historical data access information of the RAID card is collected, a historical data set is generated, and a neural network model is trained by using the historical data set to obtain a cache management model; predicting and outputting data blocks which are possibly accessed in the future and the access probability of the data blocks through the cache management model; the RAID controller dynamically adjusts a cache strategy in combination with an LRU strategy according to a prediction result output by the cache management model and a read-write request of a file system, and optimizes a storage position and an updating mechanism of a data block in a cache space; and when the hit rate or the data access efficiency does not reach the preset value, adjusting the parameters of the cache strategy and updating the cache strategy. According to the method, cache resources can be more effectively distributed, the cache hit rate is improved, the cache replacement overhead is reduced, and therefore the performance of a storage system is optimized.

An intelligent monitoring, inversion and prediction system for a transient process of pumped storage units is provided, including: an equipment layer including sensors for obtaining hydraulic, mechanical and electrical operation data of the pumped storage units in real time, a data layer including a real-time database, a relational database, an expert knowledge base and a file system and used for storing hydraulic, mechanical and electrical operation parameters collected by the equipment layer in different levels, a twin layer including a modeling and an application of a digital twin model and used for a three-dimensional modeling and a key information visual display of a pumped storage system with BIM technology and a service layer displaying one-dimensional time domain parameters and designing and visualizing front-end interfaces for six functions: operation optimization, online monitoring, inversion, dynamic prediction, health assessment and fault warning.

Embodiments are directed to a file system that include object stores. An object store for write requests may be provided. Write ahead log (WAL) entries that include data blocks may be generated. A WAL object may be generated based on the WAL entries and stored in the object store. An in-memory overlay may be updated to associate the data blocks with the WAL object. A checkpoint operation may be executed to: generate an index object that includes index entries that associate other data blocks with data objects stored in the object store; update the index object to include index entries that associate the data blocks with the WAL object; store the updated index object in the object store; update the in-memory overlay to remove the association of the data blocks and the WAL object and update the in memory WAL to remove records of successfully checkpointed WAL objects.

Serverless containerization of a collaboration application improves deployment and security of the collaboration application. A system may generate a container within a private cloud. The container may execute the collaboration application. The container is assigned an Internet Protocol (IP) address and is associated with a security group that controls network traffic to and from the container. The system may include a network load balancer configured with the IP address assigned to the container that allocates computational resources for executing the collaboration application. The system may include serverless resources such as a database and filesystem that is accessible to the container within the private cloud. The system may include a private cloud (PC) endpoint configured to establish a private connection with an end user interface, wherein the private connection provides secure network traffic between the PC endpoint the end user interface within a private network of the PC endpoint.

Embodiments are directed to managing data in a file system. In response to a request from a client of a spoke file system to access file system items shared by a hub file system: determining a portion of the spoke file systems that may have an existing lock on the file system items; obtaining log entries from the portion of the spoke file systems such that the log entries are associated with commands that were executed in the portion of the spoke file systems; executing the commands in the hub file system to update the file system items in the hub file system; granting a lock on the file system items in the hub file system that enables the client to access the updated file system items from the spoke file systems.

Embodiments are directed to a file system that include object stores. An object store for write requests may be provided. Write ahead log (WAL) entries that include data blocks may be generated. A WAL object may be generated based on the WAL entries and stored in the object store. An in-memory overlay may be updated to associate the data blocks with the WAL object. A checkpoint operation may be executed to: generate an index object that includes index entries that associate other data blocks with data objects stored in the object store; update the index object to include index entries that associate the data blocks with the WAL object; store the updated index object in the object store; update the in-memory overlay to remove the association of the data blocks and the WAL object and update the in memory WAL to remove records of successfully checkpointed WAL objects.

The invention relates to the technical field of computer storage system and file system data management, and particularly discloses an MLP-based ZNS SSD garbage collection sensing method, which comprises the following steps of: introducing an intelligent monitoring and load sensing algorithm, monitoring a system load and a data writing mode in real time, and dynamically adjusting a garbage collection strategy according to the running state of a current system, so that the garbage collection efficiency is improved. More efficient resource management and performance optimization are realized; the garbage collection operation is only carried out when the system load is relatively low, so that the interference on normal I / O operation is reduced, the overall performance of the system is maintained, and the response speed and the stability of the system are improved especially in a high-load application scene. According to the scheme, the sequential write-in characteristic of the ZNS SSD is utilized, an intelligent garbage collection sensing mechanism is combined, an efficient garbage collection algorithm is provided, and the write-in amplification effect is further reduced and the storage space utilization rate is optimized by optimizing the trigger condition, the collection priority and the system self-adaptive adjustment mechanism of garbage collection.

The invention discloses a permanent incremental backup method based on an S3 bucket-level virtual snapshot, and relates to the technical field of data storage and backup, the method comprises the following steps: initializing a backup system and starting a backup agent program, an index management service and an S3 object storage service; complete backup is executed, and a VID and S3 bucket-level virtual snapshoot are generated; efficiently managing a mapping relation between a file path and a snapshot identifier through a B + tree and an inverted index by utilizing a MongoDB index service; incremental backup is executed, and file changes are accurately recognized through Hash comparison and modification time comparison; a storage position is optimized by adopting an incremental storage and deduplication storage mechanism and an IncDSI optimization algorithm; executing backup chain integration and cleaning; and performing data recovery based on the VID of the file and the directory. According to the method, the backup efficiency is remarkably improved, the storage overhead is reduced, the data recovery process is optimized, and the method is suitable for data backup and management of a large-scale file system and has important application value and prospect.

The invention discloses a 3DTiles data storage method and a map data query method.The 3DTiles data storage method comprises the following steps that (1) an LOD hierarchy pyramid type hierarchy table is established, and an LOD range index is created; 2) generating a three-dimensional object into an agent point, and segmenting a tile according to a KD-tree structure to construct a three-dimensional KD-tree index tree; and 3) storing the data of the KD-tree leaf node into a GridFS of MongoDB multi-mode storage, mapping a file path into a unique hash value, writing the unique hash value into a corresponding node according to a KD-tree node storage structure, and creating a geographic spatial index. The method creatively solves the problems that file system management is difficult due to the fact that the number of files is large, a hierarchical structure based on a file directory is difficult to support efficient spatial range retrieval, and the spatial query response time is reduced to the millisecond level in combination with a three-dimensional KD-tree index and a MongoDB geographic spatial index.

The invention discloses a rural industry agricultural product full-process traceability method based on a block chain. The method comprises the following steps: collecting agricultural product full-life-cycle process data and environmental parameter data through Internet of Things sensing equipment; preprocessing the agricultural product traceability data by using a space-time agricultural data compression technology; storing the compressed key traceability data in a block chain network, and storing high-capacity original data through a content addressing interstellar file system; maintaining the consistency of the block chain network based on an adaptive agricultural season consensus algorithm; automatically executing data verification and business logic processing through the smart contract; the authenticity of agricultural product traceability information is verified based on a zero-knowledge proving technology, and meanwhile peasant household sensitive data are protected.

Embodiments are directed to tiered data store with persistent layers. A write tier in the file system for storing in a file system. A value for a performance metric that corresponds to write requests to the file system may be predicted based on characteristics of the write requests such that the performance metric may be determined based on a plurality of interactions with the write tier. The predicted value that exceeds a threshold value of the performance metric may be employed to cause performance of further actions, including: queuing a portion of the write requests in a memory buffer based on the predicted value and the threshold value; combining the queued portion of the write requests into s; storing the data segments in the write tier such that a measured value of the performance metric may be less than the threshold value.

Systems, methods, and software can be used to determine risks of software files. In some aspects, a method includes: obtaining an input, wherein the input comprises a binary file; determining a second set of feature vectors of the input; performing a canonical correlation analysis (CCA) on the second set of feature vectors and a first set of feature vectors to obtain a first vector and a second vector; calculating a correlation coefficient value of the first vector and the second vector; obtaining a third set of feature vectors based on the correlation coefficient value; and providing, based on the third set of feature vectors, information indicating a level of a security risk of the input and information indicating features associated with the security risk of the input.

Embodiments are directed to a file system that include object stores. An object store for write requests may be provided. Write ahead log (WAL) entries that include data blocks may be generated. A WAL object may be generated based on the WAL entries and stored in the object store. An in-memory overlay may be updated to associate the data blocks with the WAL object. A checkpoint operation may be executed to: generate an index object that includes index entries that associate other data blocks with data objects stored in the object store; update the index object to include index entries that associate the data blocks with the WAL object; store the updated index object in the object store; update the in-memory overlay to remove the association of the data blocks and the WAL object and update the in memory WAL to remove records of successfully checkpointed WAL objects.

A video gateway system at a worksite is coupled to multiple cameras on a network, and backs-up video streams generated by the cameras to a backend cloud backup video storage system and a frontend (cache) video storage system. The video gateway system generates an aggregated video asset from a plurality of streams of video from the multiple cameras, and generates metadata and a backup report associated with the video asset. The video asset, metadata, and backup report are stored on the backend cloud backup video storage system in an file system, and are also stored on the frontend (cache) video storage system such that for at least a period of time, the video asset and the associated the metadata and backup report are stored on both the cloud backup video storage system, facilitating quick access and retrieval to the stored video for retrieval for streaming, activity detection, and other uses.

Techniques are described for enabling concurrent cross-region replications and garbage collection while maintaining consistency and data integrity among file systems. In some embodiments, techniques for garbage collection fencing utilize a system-level garbage fencing key (GC fencing key) and one or more job-level GC fencing keys in a source file system that perform one or more cross-region replications with one or more target file systems, one replication and one job-level GC fencing key per target file system. In some embodiments, one job-level GC fencing key in a source file system and one job-level GC fencing key in a source file system together provide garbage fencing for a cross-region replication. In certain embodiments, the metadata information in a GC fencing key can inform, instruct, or be used to configure garbage collectors to skip garbage collection for a range of snapshots in a file system.

Technology embodied in a method that includes receiving, as an input to a machine-learning model, data indicative of user-interaction of a particular user with a generative artificial intelligence (AI) system. The machine learning model is trained to identify one or more topics associated with inputs provided to the machine-learning model. The method also includes identifying a first context associated with the data indicative of the user-interaction with the generative AI system, and parsing a file system to determine that one or more folders within the file system correspond to the first context. The file system includes multiple folders each corresponding to a separate topic as identified from historical interactions of the particular user with the generative AI system. The data indicative of the interaction is augmented and provided to the generative AI system for generation of a response to the interaction.

Techniques are described for efficient replication and maintaining snapshot data consistency during file storage replication between file systems in different cloud infrastructure regions. In certain embodiments, provenance IDs are used to efficiently identify a starting point (e.g., a base snapshot) for a cross-region replication process, conserve cloud resources while reducing network and IO traffic.

A kernel monitor can be used to mitigate ransomware activity of a host system. In some aspects, a computing system can use the kernel monitor to monitor a set of system calls generated by the host system within a time window to perform a functionality. The kernel monitor can include a respective kernel program monitoring each system call in the set of system calls. The set of system calls can be filtered by the kernel monitor to identify a subset of system calls associated with encrypting a filesystem of the host system. The computing system can determine that the subset of system calls is indicative of ransomware activity associated with the host system based on the subset of system calls exceeding a predefined threshold. Subsequently, the computing system can perform a mitigation operation to mitigate the ransomware activity.

The invention discloses a method, a product and equipment for sharing a file system by multiple hosts and a storage medium, and relates to the technical field of computers, which comprises the following steps: configuring a shared memory device as a character device, directly mapping to a user mode virtual address space, bypassing a kernel page cache, and directly reading and writing a physical address of the shared memory. A shared memory is divided into a super block area, a metadata area, a log area and a data area during formatting, all hosts perform unified operation through the shared metadata area, stand-alone cache interference is avoided, a hidden metadata file is created to map a non-data area during mounting, a log item is scanned to reconstruct a directory file structure, and the mounting efficiency is improved. All hosts generate a directory file structure completely consistent with the shared memory through log replay, the problem of data consistency caused by a cache strategy in a traditional multi-host file system is solved, and the effects that multiple hosts share a unified memory view, operation is real-time and synchronous, and data access delay is reduced are achieved.

The invention discloses a cache processing method of heterogeneous equipment, a heterogeneous system, a product, a device and a medium, and relates to the technical field of data storage. And according to the target virtual address access times corresponding to each device in the heterogeneous system, realizing a global data caching mechanism. And determining a file page corresponding to the target heterogeneous device based on the target virtual address access frequency, so that the file system layer has the capability of managing the memory cache of the heterogeneous device. And the predicted file page of the target heterogeneous device is verified, and the file page is moved under the condition that the verification is passed, so that the distribution accuracy of the target heterogeneous device is improved. The file page of the target heterogeneous device is mapped and moved to the memory of the host side, the target heterogeneous device does not need to be accessed, the file page stored in the memory of the processor is directly accessed, the increased access rate of the processor accessing the target heterogeneous device is avoided, the delay of accessing the memory of the target heterogeneous device is reduced, and the access efficiency is improved. And the data transmission rate and the system efficiency of the heterogeneous system are improved.

Techniques are described for efficient replication and maintaining snapshot data consistency during file storage replication between file systems in different cloud infrastructure regions. In certain embodiments, snapshot creation and deletion requests that occur during cross-region replications may be temporarily withheld until appropriate times to execute such requests safely, depending on the timing relationship between such requests and cross-region replication cycles.

An intelligent legal document generation and review system and associated methods are disclosed. The system comprises one or more processors coupled to memory storing instructions to provide a user interface for receiving user input, execute a natural language processing subsystem to extract relevant information, access a legal rules database, store document templates, and execute an AI algorithm. The AI algorithm retrieves a relevant template based on user input and extracted information, populates the template to generate a customized legal document, and stores the document. An attorney feedback loop allows attorneys to provide feedback that the AI algorithm learns from to improve document generation. The AI algorithm collaborates with attorneys to revise and finalize documents based on the feedback. The system streamlines document preparation, automates complex tasks, and provides expert guidance to help users adhere to legal requirements across jurisdictions.

The embodiment of the invention provides a memory recovery method and device, electronic equipment and a storage medium, and relates to the technical field of data storage. Under the condition that both the file system and the storage device support the perceptual garbage collection function, the device state of the storage device is obtained in response to the starting of the perceptual garbage collection function; according to the equipment state, determining whether an execution condition for sensing a garbage collection function is met or not; and if the execution condition of the garbage collection sensing function is met, the file system is linked to perform memory collection on the storage device, so that the file system can know the data blocks moved by the storage device, the file system is prevented from moving the data blocks moved by the storage device again, and the efficiency of the file system is improved. In addition, invalid blocks in the file system and the storage device can be synchronized, the situation that invalid data blocks in the file system are moved when the storage device executes memory recovery is avoided, and therefore the service life expenditure of the storage device and the write-in amplification value of the storage device are reduced.

Systems and methods for performing a zero-copy volume move between nodes of a distributed storage system are provided. In one example, an approach for performing a zero-copy volume move is proposed in which volume data may be maintained in place within a storage pod and need not be copied to move a given volume between the source node and the destination node. In one embodiment, metadata (e.g., a top-most physical volume block number (PVBN) of a node tree representing the volume at issue) of a write-anywhere file system is copied from the source node to the destination node. Since the storage pod is associated with a global PVBN space that is visible and accessible to all nodes of the distributed storage system, as a result of copying the top-mode PVBN of the volume to the destination node, anything below the top-most PVBN will automatically be visible to the destination node.

Embodiments are directed to workload allocation for file system maintenance. A file system that includes storage nodes and snapshots may be provided such that each snapshot may be associated with a plurality of data blocks. If snapshots are deleted further actions may be performed, including: determining the dead blocks associated with the deleted snapshots such that each dead block may be a data block that may be unassociated with undeleted snapshots; adding the plurality of dead blocks to dead trees located on the storage nodes; determining an urgency score based on a workload model and file system metrics; determining delete tasks based on the urgency score; determining a portion of the storage nodes based on a number of delete tasks; and executing the delete tasks on the portion storage nodes to delete the dead blocks to return storage capacity to the file system.

The invention discloses an air-ground cooperative vehicle networking distributed digital twin service credible sharing method, which comprises the following steps of: firstly, according to an air-ground cooperative traffic data type, performing hierarchical packaging and storage on operation data of a vehicle networking distributed digital twin system; secondly, an authoritative random equilibrium consensus algorithm is used, and a distributed edge digital twinning system of the air-ground cooperative Internet of Vehicles carries out consensus on blocks generated by data and uploads the blocks to a block chain; and finally, requesting a twinning service from the remote digital twinning system at the edge digital twinning system, and verifying the service record through a directed acyclic graph block chain and an interstellar file system, thereby realizing the trusted sharing of the service among the edge digital twinning systems in the air-ground collaborative internet-of-vehicles distributed digital twinning system. According to the method, the authenticity and tamper-proof capability of the operation data of the air-ground cooperative Internet of Vehicles distributed digital twin system in cross-node sharing are ensured, the access efficiency of large-scale data is improved, and technical support is provided for cooperation of the Internet of Vehicles and the unmanned aerial vehicle.

A method for reducing write latency in a distributed file system. A write request that includes a volume identifier is received at a data management subsystem deployed on a node within a distributed storage system. The data management subsystem maps the volume identifier to a file system volume and maps the file system volume to a set of logical block addresses in a logical block device in a storage management subsystem deployed on the node. The storage management subsystem maps the logical block device to a metadata object for the logical block device on the node that is used to process the write request. The mapping of the file system volume to the set of logical block addresses in the logical block device enables co-locating the metadata object with the file system volume on the node, which reduces the write latency associated with processing the write request.

The invention belongs to the technical field of lightning arrester disassembly and assembly, and particularly relates to an energy data trusted sharing method based on multi-block chain cooperation and improved consensus. Comprising the following steps: S1, finishing on-chain and off-chain distributed storage of data in combination with an interstellar file system and a block Hash; s2, generating publishable data fingerprints by combining the smart contract and row data; s3, performing data flow sharing based on the publishable energy data of the block chain; s4, performing a consensus based on POBT +; s5, finishing the uplink evidence storage of the data circulation process by combining the smart contract; s6, carrying out abnormal behavior identification; s7, data circulation tracing is carried out; through the steps, access control and abuse responsibility investigation are realized in combination with attribute encryption and digital fingerprint technologies, construction of a security environment in the data sharing process is greatly improved, flexible and revocable fine-grained access control is realized, accurate responsibility investigation of unauthorized use of data is completed, and high efficiency and security in the energy data sharing process are ensured.