407 results about "Invalid Data" patented technology

The invention discloses a real-time interaction and intelligent management method for a multi-dimensional data space, and relates to the technical field of data interaction management, and the method comprises the following steps: I, collecting multi-source data, including structured data, unstructured data and spatio-temporal data, in real time through a diversified collection terminal, and dynamically adjusting the sampling frequency based on a data entropy value, carrying out cross-device space-time synchronization calibration and edge end lightweight processing on the acquired data; iI, preprocessing the data flow in real time, fusing the multi-modal data, and the like. According to the method, the sampling frequency is dynamically adjusted based on the data entropy value, and in combination with edge end lightweight processing, invalid data transmission is reduced, and the key data acquisition efficiency is improved; hot data is stored in a time sequence database and a memory database, real-time monitoring of an emergency scene and second-level response of emergency command are supported, dynamic capacity expansion and shrinkage of computing resources are matched, edge-cloud end collaboratively distributes tasks, network delay is reduced, and system stability and response speed in a high-concurrency scene are guaranteed.

The invention discloses an MQTT message transmission optimization method and system. The method comprises the steps of analyzing a theme, extracting a device type, a data feature and a geographic position triple, calculating a hash value, and mapping a device to a specified Broker fragment cluster node to generate a fragment mapping table; processing the equipment data in the edge domain in the fragment mapping table through an edge calculation layer, removing invalid data according to a preset rule, merging the equipment data in the same fragment node, and embedding a fragment node ID for an aggregation message generated after merging; identifying a fragment node ID and routing to a target fragment node, positioning a corresponding shared memory pool, and writing the message into the shared memory pool; and responding to a direct access request of the client, so that the client directly accesses the data from the shared memory pool through the user mode network stack. According to the method, the problem of uneven load is effectively solved, multiple times of state switching in the data transmission process is avoided, and the MQTT message transmission efficiency is improved while the transmission cost is reduced.

The invention provides a multi-source log fusion driven database monitoring method and system, and relates to the technical field of computers.The method comprises the steps that format analysis processing is conducted on multi-source heterogeneous logs of a to-be-monitored database on the basis of a dynamic analysis template library, and initial database logs are obtained; performing data preprocessing on the initial database log based on a rule engine to obtain a database log; and performing streaming processing on the database log to obtain a log event stream, and sending the log event stream to the server side, so that the server side detects the log event stream based on the log anomaly detection model and the associated alarm analysis engine, and generates a monitoring analysis result of the to-be-monitored database. By means of the mode, the technical bottleneck that heterogeneous database logs are difficult to analyze is broken through, the problem that the terminal is poor in database log format compatibility can be solved, invalid data transmission between the terminal and the server side is effectively reduced, and the database monitoring effect is improved.

The application discloses a safety production monitoring data management method and device, equipment and a storage medium, and comprises the following steps: acquiring real-time production data in the production process of an enterprise in real time and storing the real-time production data into a preset database; extracting multi-source time sequence data from the preset database to generate an original data set; screening out data that does not conform to a preset rule in the original data set to generate a primary selected data set; according to the strong time correlation characteristics of the dangerous chemical production data in the primary selected data set, eliminating invalid data through a preset validity rule to generate an effective data set; periodically extracting data from the effective data set, and displaying the management result of the time sequence data in a trend chart through a preset visual display model. The application can improve the accuracy of analysis results when analyzing safety production monitoring data.

The invention discloses a data forwarding processing method and system for a campus network, and the method comprises the steps: controlling a plurality of independent cameras to generate a multicast video stream carrying a permission identifier according to an access control terminal triggering event; the method comprises the following steps: synchronously creating (S, G) table entries with permission marks and never expired (*, G) table entries on a path multicast router according to a first video stream data packet received by a multicast source side designated router source DR; according to an IGMP join message which is sent by a receiver and carries a permission identifier, through a matching result of an RP comparison (*, G) table entry output interface permission and a multicast data permission identifier, whether a data stream is forwarded along a shared tree (RPT) is decided; and according to a matching result of the permission identifier of the new receiver and the pruned (S, G) flow permission, activating a pruning state on the multicast path and recovering data flow forwarding. According to the embodiment of the invention, occupation of invalid data streams on campus network bandwidth can be reduced, and the overall utilization rate of network resources is improved.

The invention discloses a garbage collection optimization method and device, equipment and a medium. The invention relates to the technical field of solid state disks, which comprises the following steps that: a global invalid data bit chart is established in a host memory, each table item in the global invalid data bit chart corresponds to a physical block, and a bitmap bit in each table item corresponds to a physical page in the physical block; bit values corresponding to the bitmap bits comprise a first bit value and a second bit value; responding to a data updating event through an access acceleration engine so as to dynamically update the bit value of the corresponding physical page in the global invalid data bit graph; and when garbage collection is triggered, screening the physical page of which the bit value is the second bit value in a to-be-collected source physical block according to the global invalid data bit chart and the access acceleration engine to perform validity verification and data migration, and skipping the physical page of which the bit value is the first bit value. According to the invention, the garbage recycling efficiency is improved.

The invention relates to the technical field of communication security, and particularly discloses a communication terminal information acquisition method and system. The method comprises the following steps: acquiring an original data stream of a communication terminal, and generating a task queue through data source marking, dynamic load balancing and protocol packaging; performing desensitization, metadata annotation and lightweight encryption on the task data to form an encrypted data block; secure transmission is realized through transmission channel selection and quality monitoring; decrypting the received data to generate a time sequence database record; generating a rule update package based on invalid data detection and machine learning analysis; and finally, outputting a standardized data set through filtering and verification. According to the invention, efficient acquisition, secure transmission and intelligent processing of mass communication data are realized, and the real-time performance and accuracy of data processing and the load balancing capability of the system are effectively improved.

The invention belongs to the technical field of intelligent control, and discloses a robot visual interaction control method and device, electronic equipment and a storage medium, when the electronic equipment is started, an ROS2 communication environment is initialized, and a Qt graphical user interface is loaded; whenever any user interface subpage on the Qt graphical user interface is detected to be switched from the hidden state to the display state, adding one to the count value; when any user interface sub-page is detected to be switched from the display state to the hidden state, subtracting one from the count value; when the count value is changed from 0 to 1, starting ROS2 topic subscription; when the count value is changed from 1 to 0, the ROS2 topic subscription is canceled; by introducing a dynamic binding mechanism of the interface visibility and the ROS2 subscription state, the data subscription is automatically cancelled when the interface component is hidden, so that the continuous receiving and processing of background invalid data streams can be avoided, the CPU resource occupancy rate and communication resources are effectively reduced, the communication load and processing overhead are further reduced, and the operation fluency and response speed are improved.

The invention discloses a non-intrusive image capturing and storing device and a non-intrusive image capturing and storing method. Relates to the technical field of security data storage, and solves the problems of acquisition blind areas, low data credibility and invalid data redundancy caused by a traditional software-level screenshot technology. The video conversion processing module is directly connected with a target host video output link through a standard video interface, the whole device is independent of a host system, does not depend on host permission, does not perform data interaction or protocol communication with a host, and avoids a collection blind area caused by insufficient permission or protection interception; the control processing module encrypts the frame-cut image to protect data security; the control processing module only triggers frame interception when detecting that the image is obviously changed, so that undifferentiated acquisition is avoided, redundant data irrelevant to auditing is reduced from the source, and the storage and operation cost of subsequent auditing analysis is reduced; the protection operation of the safety protection module is executed through a hardware interface and does not depend on host software logic, so that protection failure caused by remote control of a host is avoided.

The invention relates to the technical field of data processing, in particular to a real-time data aggregation optimization method and system based on incremental calculation. The invention discloses a real-time data aggregation optimization method and system based on incremental calculation. The method comprises the following steps of data receiving and preprocessing, incremental calculation model construction, data aggregation and storage, data updating and maintenance, and query and analysis. Data receiving and preprocessing: receiving data streams from a plurality of data sources in real time, performing cleaning operation on the data streams to remove invalid data, performing format conversion to unify a data format, and allocating a unique identifier and a record timestamp to each processed data record; according to the method, only the change part of the data needs to be calculated through the incremental calculation model, so that the calculation amount is greatly reduced, and the efficiency of real-time data aggregation is improved; compared with a traditional method and system, the calculation time can be shortened by more than 50% when large-scale real-time data are processed.

The invention provides a federal retrieval enhancement generation method oriented to multiple data sources. The method comprises the following steps: initializing the reliability of federal data sources; dividing a user query field to obtain the reliability of the federal data source to the user query field; selecting a target data source from the federated data source; performing local retrieval on the target data source to obtain candidate answers; estimating the confidence of the candidate answers; according to the confidence coefficient, filtering the candidate answers, and generating a user query answer; and according to the confidence, updating the reliability of the federal data source, and performing confidence estimation on the candidate answers, so that a confidence estimation result has rich semantic information and can be well identified and utilized by a large language model, thereby being beneficial to solving the conflict problem between exogenous knowledge and model endogenous knowledge, and improving the reliability of the model. And through federal data source quality detection of feedback iteration, reliability evaluation of the data source is continuously converged to the domain knowledge accuracy, efficient and accurate detection of the data source quality is realized, and query of invalid data sources and low-quality data sources is effectively reduced.

The invention provides a sparse adaptation circuit and a network-on-chip sparse data processing method, the sparse adaptation circuit is embedded between a network interface unit (NIU) and a router of an NoC system, and the sparse adaptation circuit comprises an input processing module, a compression engine and an output processing module; the sparse detection unit is used for performing real-time zero value detection on the data stream received from the network interface unit NIU to generate sparse identification information; the compression engine is coupled with the sparse detection unit and is used for carrying out dynamic compression coding on the data stream based on the sparse identification information; and the output interface is used for sending the processed data flow to a router in a format conforming to the network-on-chip protocol. In the network-on-chip, a hardware method is adopted to identify a data packet containing a large number of zero values in AI calculation and compress the data packet into a data packet only containing valid data, so that the problems of low transmission efficiency, waste of power consumption and the like caused by the fact that on-chip bandwidth is occupied by invalid data are avoided.

The invention belongs to the technical field of intelligent humanoid robots, and discloses a remote operation data acquisition and data closed-loop system for an intelligent humanoid robot, which builds an instruction, action and acquisition ternary time sequence synchronization mechanism through a time sequence coordination module, realizes dynamic coordination of the three parts by using a time sensitive network technology, effectively reduces time sequence deviation, and improves the system performance. Meanwhile, a space-time alignment algorithm of the data processing module is combined, so that the consistency of the multi-modal data in time and space dimensions is guaranteed; acquired data can accurately fit human operation logic and a robot action state, the optimization efficiency of a data closed loop is improved, and the requirements of high-precision operation scenes such as complex assembly and flexible material processing are successfully met; a quantitative association system of data quality and control effects is established by means of the evaluation module, invalid data such as sensor faults and action fuzziness can be screened in real time, and occupation of redundant storage and computing resources is reduced.

The invention provides an electric power communication SDH network operation state real-time monitoring system and method, and the system comprises a multi-protocol adaptive collection module which carries out the collection of network element information, performance information, alarm information and equipment information of an electric power communication SDH network according to a preset strategy, performing standardization processing on the acquired multi-source heterogeneous data through a protocol analysis engine module to obtain structured data; the intelligent data screening and deconstruction module is used for analyzing, associating and identifying the structured data to obtain a key business entity; the lightweight packaging module is used for carrying out serialization and compression optimization on the key business entities to obtain lightweight packaging data; and the rendering module is used for carrying out real-time rendering on the basis of the lightweight packaging data to obtain a real-time visual operation state of the electric power communication SDH network. According to the invention, the acquisition efficiency is optimized, real-time monitoring is supported, the bandwidth cost is reduced, invalid data transmission is reduced by 80%, and the fault positioning time is compressed from an hour level to a minute level.

The invention discloses a harmonic traceability method and device based on power grid topology, a terminal and a medium, and the method comprises the steps: collecting multi-source data, tracking a harmonic back propagation path, and screening key nodes; collecting equipment features, waveform features and time sequence features based on the key nodes, constructing a harmonic source feature library, and performing per unit value conversion on harmonic power features of the key nodes; calculating the matching degree between the key node and the target type equipment in the feature library by using the harmonic source feature library, and determining a harmonic source node and a potential harmonic source node according to the matching degree; calculating the contribution degree based on the per-unit value, and determining a target harmonic propagation path and a target harmonic source on the path; according to the method, a harmonic propagation path is accurately tracked through double criteria, key nodes are screened, and invalid data processing is reduced; meanwhile, the influence weight of each node is determined through hierarchical calculation of contribution degrees, the traceability precision is effectively improved, and the harmonic risk of the power grid is reduced.

The invention relates to the technical field of big data processing, and discloses a cloud computing and edge computing-based big data co-processing method and a cloud server, which are applied to a co-processing system consisting of an edge node cluster and the cloud server. And obtaining target data through a preprocessing strategy consisting of data filtering, format conversion, increment extraction and lightweight compression. According to the method, the compression ratio larger than or equal to 62% is achieved in combination with the improved LZ4 algorithm, only differential data with service value is transmitted, the invalid data transmission amount is greatly reduced, and the network bandwidth occupation pressure is remarkably reduced; and a dynamic task scheduling link, which is fused with a genetic-reinforcement learning hybrid algorithm of bidirectional LSTM load prediction, can accurately predict edge node computing power fluctuation, dynamically divide task types, match optimal processing nodes, control real-time task response delay within a reasonable range, and realize maximization of computing power resource utilization rate at the same time.

The present application belongs to the technical field of embodied intelligent robots, and discloses a kind of embodied intelligent humanoid robot remote control data acquisition and data closed loop system, and the instruction, action, acquisition ternary time sequence synchronization mechanism is built by time sequence cooperation module, the dynamic cooperation of three is realized using time sensitive network technology, effectively reduce time sequence deviation, while combining the space-time alignment algorithm of data processing module, guarantee the consistency of multi-modal data in time and space dimension;Make the collected data can accurately fit human operation logic and robot action state, improve the optimization efficiency of data closed loop, successfully adapt to the demand of complex assembly, flexible material processing and other high-precision operation scene;Rely on evaluation module to establish the quantitative correlation system of data quality and control effect, can real-time filter invalid data such as sensor failure, action fuzzy, reduce the occupation of redundant storage and computing resources.

The application discloses a data query method, device and equipment and a storage medium, and comprises the following steps: when it is identified that a specified element is included in a query statement SQL, a PartitionTopN operator is added at a specified position of an initial plan to obtain a latest plan; original data obtained by executing the latest plan is filtered by using the PartitionTopN operator to obtain effective data; and a query result of a database is determined according to the effective data obtained after filtering. The PartitionTopN operator is added in the initial plan corresponding to the initial query statement SQL, so that invalid data is filtered in advance when the latest plan is executed, the amount of data that needs to be processed by a window operator is greatly reduced, the use of memory and the occupation of CPU resources are reduced, the initial query statement is not changed in the optimization process, the results before and after optimization are consistent, and the efficiency of data query and the overall performance of the system are improved.

The invention relates to a communication method, terminal equipment and a base station. The method comprises the following steps: counting a filling frame containing downlink data within a preset duration, wherein the filling frame is a sub-protocol data unit containing filling bits; judging whether the downlink air interface is congested or not according to the counted filling frames; and under the condition that the downlink air interface is not congested, the TCP connection is reestablished. Wherein the filling bits are invalid data occupied bits and represent idle resources, and according to the embodiment of the invention, the condition of the idle resources can be obtained by counting the filling frames, and whether the downlink air interface is congested or not can be judged. For example, if idle resources exist or many idle resources exist, the possibility of downlink air interface congestion is small, TCP connection is reestablished, normal data transmission can be quickly recovered under the condition that TCP data transmission is slow, the lagging time of data transmission is reduced, smooth communication is guaranteed as much as possible, manual operation of a user is not needed, and user experience is improved.

The invention relates to the technical field of Internet of Things, and particularly discloses an Internet of Things data acquisition method and system, and the method comprises the steps: constructing a three-stage data acquisition architecture composed of a main control node, edge acquisition nodes and terminal sensing equipment; the terminal sensing equipment dynamically adjusts own data acquisition frequency based on a built-in activeness evaluation module; the edge acquisition node receives the data sent by the terminal sensing device and executes distributed data verification and data compression processing; the main control node schedules the processed data uploaded by the edge acquisition node according to the network load condition and the data priority; and the main control node performs aggregation analysis on the received data, and optimizes the acquisition strategy of the whole network based on the analysis result. The acquisition frequency can be automatically adjusted according to the equipment state and the data change, invalid data acquisition is reduced, and the equipment energy consumption is reduced. According to the distributed data verification mechanism, the integrity and non-tampering property of the data are ensured, and the credibility of the data is improved.

According to one embodiment, a memory system includes a controller. The controller manages a plurality of flags corresponding to a plurality of physical addresses. In response to receiving a read command from a host, the controller acquires from a first management table a physical address mapped to a logical address specified by the read command, and reads data from a nonvolatile memory based on the acquired physical address. The controller determines whether or not to transmit the read data to the host based on whether a current status of the flag corresponding to the acquired physical address is set to a first value indicating valid data or to a second value indicating invalid data.

A data management method, apparatus, device, a computer-readable storage medium, and system are provided. The method is applied to a first network device, the first network device determines target invalid data existing in a second network device, the first network device obtains a first RPC, and the first RPC carries information about target invalid data that needs to be deleted. Then, the first network device sends the first RPC to the second network device, where the first RPC is used to indicate the second network device to delete the target invalid data. In this application, residual data is automatically identified, and the residual data is cleared, to automatically delete the residual data in batches, thereby saving device resources and improving an operation and maintenance management effect.

Embodiments of the present application disclose a database modification method, device, equipment and medium. The method comprises: when a transaction lock object of a target transaction is released and a waiting transaction is woken up, determining registration of lock information of a target record of the waiting transaction in a conflict transaction registration table; determining an execution strategy of the waiting transaction according to the registration; and controlling the waiting transaction to modify data according to the execution strategy. According to the embodiments of the present application, when the transaction lock object of the target transaction is released and the waiting transaction is woken up, the registration of the lock information of the target record of the waiting transaction in the conflict transaction registration table is determined, and the execution strategy of the waiting transaction is determined according to the registration, so that invalid data query and data modification of the conflict transaction after the conflict transaction is woken up at the same time are avoided, and the efficiency of concurrent execution of the system is improved.

This invention relates to an automatic gear shifting control method, system, storage medium, and computing device for mining trucks. The method includes: acquiring offline state data of the mining truck and preprocessing it; sequentially generating an original MTD set; filtering out irrelevant and invalid data to generate an effective MTD set, and randomly dividing it into a training set and a test set; using the training set as input to a pre-established multi-parameter and time-span ResNet-Bi-LSTM-Attention network to train the ResNet-Bi-LSTM-Attention network; inputting the test set into the trained ResNet-Bi-LSTM-Attention network to verify its robustness, obtaining the final ResNet-Bi-LSTM-Attention network model; inputting the acquired offline state data of the mining truck into the model; outputting the final gear position feature; and completing the gear shifting operation. This invention solves the problem in existing technologies that easily overlook the integrity of temporal data context information and cannot effectively integrate the features before and after the data, and can be applied in the field of heavy vehicle gear shifting control.

This application belongs to the field of computer storage technology, specifically disclosing a cross-rack sensing invalid data reclamation method and apparatus based on erasure coding. The method includes: determining the cross-rack update network traffic generated by simulating the reclamation of invalid data blocks in each erasure coding stripe based on the invalid data blocks and valid check blocks stored in each rack; dividing each erasure coding stripe into filling stripes and reclamation stripes based on the cross-rack update network traffic corresponding to each erasure coding stripe; migrating the valid data blocks of each reclamation stripe to the storage nodes where the invalid data blocks in each filling stripe are located, and releasing the storage space of each reclamation stripe; and updating the valid check blocks of each filling stripe. This application achieves a more scientific and efficient invalid data reclamation method, effectively reducing network traffic throughout the invalid data reclamation process, improving invalid data reclamation efficiency, while ensuring system storage efficiency.

The invention belongs to the technical field of Internet of Vehicles communication, and discloses a heavy diesel vehicle remote monitoring method and system based on edge computing, and the method comprises the steps: constructing a rule engine according to a predefined transmission rule, and deploying the rule engine to an edge computing node; the data stream of the heavy diesel vehicle is transmitted to an edge computing node, a rule engine is used for evaluating each data field in the data stream, a corresponding transmission level is distributed, and the transmission level comprises a first-level field, a second-level field and a third-level field; and matching a preset transmission strategy according to the data field with the allocated transmission level, executing differentiated data processing, and transmitting a processing result to an enterprise cloud platform through a network transmission layer. According to the method, low-value data is filtered in real time through edge calculation, the invalid data uploading amount is reduced, and data transmission is reduced; in addition, through a high compression ratio and a delay transmission technology, the bandwidth occupation of low-value data is reduced, and the utilization efficiency of network resources is optimized.

The invention discloses a production scheme selection method, device and system, a storage medium and a computer program product. The method comprises the steps of obtaining multi-dimensional production data in a production process; processing the multi-dimensional production data to remove invalid data and unify data formats; target machine type information is received, and at least one corresponding initial production scheme is called from a preset production scheme library according to the target machine type information; based on a preset rule system which can be dynamically adjusted independent of a business process and contains inter-rule association logic and scenarized rule grouping, performing matching screening on the multi-dimensional production data and the initial production scheme to obtain a candidate production scheme; performing quantitative evaluation on the candidate production schemes by adopting a preset multi-dimensional evaluation index system, and determining an optimal production scheme; and verifying the optimal production scheme, and outputting the scheme after the verification is passed. According to the scheme, the flexibility and maintainability of the scheme selection technology are improved, and the scheme selection accuracy is improved.

The invention relates to a big data information collection method. The method comprises the following steps: collecting retrieval data, browsing data, consulting data and auxiliary data of multiple users in an application system; invalid data removal and noise data filtering are carried out on the collected data, invalid data removal comprises null value and meaningless data removal, duplicate data de-duplication and format standardization processing, and noise data filtering comprises regular filtering and statistical filtering; and constructing an invalid and noise data judgment model based on the processed valid data and invalid and noise data. According to the method, through data preprocessing and double screening of invalid and noise data judgment models, invalid data and noise data can be effectively eliminated, so that the accuracy of data information collection is improved; through automatic judgment of invalid and noise data judgment models and an incremental updating mechanism, the models are continuously optimized along with data accumulation, the manual screening workload of data information collection can be reduced, and the misjudgment rate of automatic judgment is reduced.

The embodiment of the invention provides an airborne 1394 bus data collection method and device, a storage medium and electronic equipment, and relates to the field of avionics system data collection, and the method comprises the steps: collecting initial data based on an airborne 1394 bus; dynamically loading a filtering rule and a parameter extraction rule; filtering the initial data based on the filtering rule to obtain target data; and extracting valid data from target data based on the parameter extraction rule. The method can solve the problems that a traditional collection mode can only statically and completely collect bus data, a large amount of invalid data exists, and follow-up storage resources and analysis processing computing power are wasted.