46 results about "Data scraping" patented technology

The invention discloses an intelligent public opinion research and judgment method and system for public security actual combat, and the method achieves the high-throughput data capture and real-time public opinion inflow through multi-source crawler, breakpoint resume, incremental collection and asynchronous concurrent processing, and greatly reduces the workload of manual screening. A unified, quantifiable and structured public opinion risk grading standard is established, and risk grading standard unification is realized by constructing three types of structured public opinion dictionaries of event subjects, event properties and sensitivity grades; the grading rule is quantified and objective and adapts to the grading system of the public security actual combat, and the subjectivity and inconsistency of research and judgment are remarkably reduced; and the recessive public opinion recognition capability is improved through the deep semantic model. According to the method, the BERT model, keyword vector fusion and the police affair large model are combined, so that accurate analysis of complex semantics such as metaphor, black talk and irony mood is realized; high-robustness semantic matching and duplicate removal are carried out; through accurate extraction of risk factors and event identification, the problem of high missed judgment rate in the prior art is improved.

The invention provides an image data processing method and device, equipment, a storage medium and a program product, and the method comprises the steps: responding to a first display interrupt signal arriving according to a preset period and a data capture request, determining a state of a coding operation of first image data triggered based on a second display interrupt signal arriving according to a preset period, the second display interrupt signal arriving earlier than the first display interrupt signal, and the first image data being image data of a first frame stored in a frame buffer queue before the second display interrupt signal arrives; triggering a third display interrupt signal in response to determining that the encoding operation of the state representation for the first image data is completed; and based on the third display interrupt signal, obtaining second image data currently stored in the first frame from the frame buffer queue, and processing the second image data. According to the method, the display interruption can be dynamically adjusted, so that the coding efficiency can be improved, and the frame loss ratio can be reduced.

The application provides an automobile IBC product offline appearance identification detection system and method, and relates to the technical field of automobile part detection. The system comprises: a feeding vision module for identifying the placement posture of the IBC product to be detected on the conveying line and outputting posture data; a mechanical hand module for grabbing the product according to the posture data and transferring it to a detection station; a rotary table module arranged at the detection station for fixing and rotating the product; a vision detection module for scanning the surface of the product during rotation, acquiring three-dimensional image data, and performing defect detection and classification based on the three-dimensional image data; a data tracing module in communication with a central control system and an enterprise MES system for binding the unique identification of the product and the detection data; and the central control system for coordinating the operation of each module and controlling the mechanical hand module to perform a sorting and stacking operation corresponding to the result according to the result of defect detection and classification. Full automation and full coverage of the detection process are realized.

The invention discloses a perimeter intrusion monitoring method and system for a tunnel portal high-speed rail line, and relates to the technical field of intelligent monitoring, and the method comprises the steps: obtaining a line tunnel portal sequence driving data set according to a high-speed rail line advancing tunnel portal sequence of a sorted target high-speed rail line, and carrying out the intrusion risk recognition, obtaining a tunnel portal sequence intrusion risk coefficient set; meanwhile, interference analysis is carried out on a high-speed rail line advancing tunnel portal sequence, a tunnel portal sequence interference factor set is obtained, a tunnel portal sequence intrusion risk coefficient set is corrected, and a preset video data capture frequency is determined based on the corrected tunnel portal sequence intrusion risk correction coefficient set; and then differentiated monitoring analysis is carried out on the high-speed rail line tunnel portal, and tunnel portal perimeter intrusion information is determined for early warning management and control. The technical effects that differentiated intelligent monitoring and analysis of the tunnel portal are achieved, the accuracy of intelligent monitoring and analysis and the processing efficiency of intrusion monitoring data are improved, and then the timeliness of early warning of the perimeter intrusion of the tunnel portal is ensured are achieved.

The invention discloses a multi-protocol data capturing method and system, and relates to the technical field of data capturing. The multi-protocol data capturing method is applied to the multi-protocol data capturing system, and the multi-protocol data capturing system comprises a client and a server which is in communication connection with the client through multiple network protocols. The multi-protocol data capturing method comprises the following steps: a client sends a data capturing request to a server through one of a plurality of network protocols; the server calls a corresponding processor to perform data capture based on the request type carried in the data capture request; the server converts the grabbed data into a unified intermediate format; and the server converts the data in the unified intermediate format to the client through one of the plurality of network protocols and sends the data to the client. According to the multi-protocol data capturing method and system disclosed by the invention, cross-language and cross-protocol data capturing service can be realized.

The invention discloses an automatic processing system and method for electric power information communication scheduling operation data. The system comprises a man-machine interaction module, a data capture module, a report processing module, a file management module and a data verification module. The man-machine interaction module is used for realizing data interaction between the system and an operator; the data capturing module is used for automatically extracting work order data through a cross-communication scheduling core business system; the report processing module is used for realizing monthly report updating, daily report generation and end-of-month data archiving; the file management module is used for managing template files, result files and log files related to the system; and the data verification module is used for verifying the accuracy of the data processing result. The method disclosed by the invention is high in data processing efficiency, good in integrity and relatively high in accuracy.

The application provides a BOP data acquisition EWI generation method, medium and system, and belongs to the technical field of production business data.The application firstly constructs a multidimensional BOP data load matrix, accurately calculates the contribution value of each parameter through a thermodynamic entropy equation, then establishes a standard EWI basic matrix, and optimizes the BOP data acquisition strategy and calculates the data integrity score by using the minimum spanning tree algorithm.In the data processing stage, the EWI mergable index is accurately generated by using the Lagrange multiplier method, the transverse error is calculated by using the Euclidean distance function to analyze the spatial relationship between parameters, the longitudinal error is calculated by using the Fourier transform to monitor the time change trend of parameters, the two kinds of errors are comprehensively formed into a hybrid error evaluation result, and finally, the comprehensive index accurately reflecting the production quality state is output in combination with the EWI mergable index.

The invention relates to an analysis method and device for a chip. The method comprises the steps that data capture is conducted on latest results of a target chip by different analysis tools to obtain target data, the number of the analysis tools is at least two, and analysis conducted on the target chip by the analysis tools is different; taking the target data corresponding to each analysis tool as an analysis basis and sending the analysis basis to other analysis tools; performing anomaly evaluation based on the target data to obtain an evaluation result; under the condition that the evaluation result is abnormal, the abnormal evaluation result serves as an analysis basis for optimization and is sent to a first tool in the analysis tools, and all the analysis tools are controlled to end analysis until an end condition is met; the end condition is that the evaluation result is normal. The accuracy and speed of chip analysis are remarkably improved, multi-dimensional linkage analysis is achieved, and the method is suitable for multiple research and development stages of chip development and wide in application range.

The invention discloses a mechanical arm grabbing anti-sliding control method, and relates to the technical field of intelligent sorting and recycling, and the mechanical arm grabbing anti-sliding control method comprises the following steps: collecting visual information of an object in real time, and determining grabbing information of a clamping jaw on a mechanical arm based on the visual information; wherein the visual information comprises displacement data, three-dimensional shape data and anti-skid data, and the grabbing information comprises a grabbing position, grabbing force, a clamping jaw opening and closing angle and a mechanical arm tail end attitude angle. Object characteristics are accurately described through displacement data, three-dimensional shape data and anti-slip data, adaptive grabbing information is output in combination with a machine learning model, and the slip risk is reduced from the source; a secondary grabbing mechanism with the preset sliding path matched with the real-time sliding path is designed according to the sliding risk, the multiple preset sliding paths are dynamically simulated, the optimal preset sliding path is dynamically matched in combination with the real-time track, rescue grabbing of the sliding object can be achieved, and the grabbing success rate can be increased.

This application discloses an adaptive data crawling method, device, storage medium, and apparatus, relating to the field of Internet technology. The method includes: configuring request parameters based on an interface request initiated by a target user, a preset configuration file, and a preset custom mapping rule table to obtain data configuration information; performing pre-crawling verification on the data configuration information to obtain pre-verification results; and matching a target-level thread pool from a preset-level thread pool based on the pre-verification results and a preset sliding window algorithm, and activating threads in the target-level thread pool to complete data crawling. Compared to traditional data crawling methods that require manual intervention for system updates and maintenance, affecting the accuracy and success rate of data crawling, this application can automatically map request parameters, notify the custom mapping rule table to assemble complex parameters and perform pre-verification of parameters, and filter thread priorities to complete data crawling, thereby improving the accuracy and success rate of data crawling.

This invention relates to a method, apparatus, equipment, and medium for automatic configuration and cost calculation of miscellaneous charges. The method includes constructing a multi-dimensional standardized template library and completing import verification to form a set of callable pricing rules; upon receiving a calculation request, retrieving and preprocessing order-related data across systems; filtering candidate rules through a rule matching engine and determining the optimal billing basis by weighting and sorting according to a preset priority system; adapting the billing mode and formula based on this basis, initiating batch parallel calculations and handling anomalies synchronously; and supporting multi-format export and cross-system synchronization after integrating the results. This invention includes modules for template library construction and verification, data retrieval and preprocessing, rule matching and sorting, cost calculation and anomaly handling, and result integration and synchronization. It achieves automatic configuration and efficient calculation of miscellaneous charges, improves data accuracy and process closure, adapts to the needs of multiple scenarios throughout the warehousing and logistics process, and significantly reduces labor costs.

The invention relates to the technical field of data detection, in particular to a real-time anomaly detection method based on multi-modal data fusion, and the method comprises the following steps: S1, constructing a data capturing platform, obtaining data information of different data types through the data capturing platform, preprocessing the data information, and generating modeling data corresponding to the different data types; s2, performing multi-scale feature extraction on the generated multi-modal modeling data to obtain three-dimensional point cloud data and data features, dynamically calculating a mutual relation between the three-dimensional point cloud data and the data features based on a cross-modal collaborative attention mechanism, and performing dynamic feature reconstruction on the three-dimensional point cloud data and the data features to obtain three-dimensional point cloud data; and performing multi-modal fusion on the reconstructed three-dimensional point cloud data and the data features to obtain final fusion features, thereby effectively solving the problems that an existing real-time anomaly detection method can only perform detection according to single data, the detection accuracy is low, an anomaly root cannot be quickly positioned during detection, and the working efficiency is low.

The invention relates to the technical field of data capture, in particular to a platform development data capture method and system based on SCADA (supervisory control and data acquisition). Comprising the following steps: S1, capturing historical data and real-time data by using an SCADA system, obtaining a data storage module corresponding to the historical data, connecting a cloud platform, and transferring the real-time data to the cloud platform for distribution; real-time frequency and timing frequency data are distinguished, effective time is set for timing data, report deadline is determined based on the shortest remaining effective time, it is avoided that expired data are used to affect an analysis result, and then missing data types needed for generating a complete report are actively recognized, so that the report deadline accuracy is improved. And before the target capture time, the cloud platform instructs the SCADA system to directionally capture the missing data, so that data completion and report generation are ensured to be completed before the deadline, and the problem that the report is lagged or incomplete due to incomplete data acquisition and aging failure in an industrial scene is solved.

The embodiment of the invention discloses a data capturing method and device, a medium and a product. The method comprises the steps that an interface combination and exclusive data blocks of all interfaces in the interface combination are determined, the interface combination comprises at least two interfaces, the at least two interfaces are complementary on the data block downloading speed level, and an exclusive data block combination corresponding to the interface combination comprises all data blocks of a target file; corresponding data blocks in a target file are downloaded through the interfaces in the interface combination, and the identifier of each data block carries a file identifier; and for each file identifier, when it is detected that all the data blocks under the file identifier are downloaded, determining a target file for the file identifier according to all the data blocks under the file identifier. According to the technical scheme provided by the embodiment of the invention, the file downloading speed can be improved under the condition that the interface transmission bandwidth is not changed.

This invention discloses a fault identification and alarm system and method for silicon steel furnace rolls based on the gradient boosting tree method. Based on the gradient boosting tree method, it utilizes furnace roll operation data within the silicon steel furnace, including time-series data and dimensional space data on rotational speed and current, to accurately identify, classify, and alarm for faults such as slight bending, heavy bending, and carbon sleeve wear of the furnace rolls. The steps include capturing furnace roll data from the existing IBA system, analyzing and filtering data in the frequency domain using dimensionality algorithms and existing AI models, and identifying and alarming the operating conditions. This invention extracts data by adding a developed plug-in to the IBA PDA, and through analysis, statistical calculation, and fault analysis of this data, it provides a direct understanding of the furnace roll's performance, technical level, and potential improvements, and triggers pre-defined alarm events to alert workers.

This invention discloses an automatic extraction method for subway train passing data based on time-frequency features, relating to the field of rail transit monitoring. The method includes: collecting subway track vibration signals and performing wavelet transform to obtain a two-dimensional time-frequency map; constructing a wheelset recognition plugin based on a YOLO model to recognize the two-dimensional time-frequency map and obtain initial wheelset recognition results; obtaining a timestamp interval sequence, calculating the coefficient of variation of the timestamp interval sequence, filtering initial wheelset recognition results with a coefficient of variation greater than or equal to a threshold to obtain a negative sample set, and using the remaining initial wheelset recognition results as a multi-train sample set; performing feature extraction based on a random sample consensus algorithm to obtain a positive sample set; performing transfer learning on the wheelset recognition plugin to obtain an optimized wheelset recognition plugin, performing wheelset recognition, and calculating and obtaining subway train passing data. This method solves the problems of inconsistent data acquisition standards, non-standard processing procedures, low overall efficiency, and insufficient reliability in existing technologies.

The invention discloses a distributed garden ecological data acquisition system based on LoRa, and belongs to the technical field of data acquisition. The distributed garden ecological data acquisition system based on LoRa comprises a data acquisition module, a data transmission module, a data processing module and a data storage module. The problems that in the prior art, data capture is incomplete or interrupted possibly, manual collection efficiency is low, and large-scale data collection requirements are difficult to meet are solved, data collection is carried out through the LoRa terminal nodes, data transmission levels are reduced, data collection and transmission efficiency is improved, the overall reliability of the system is improved, and the system is suitable for popularization and application. Even if some nodes temporarily lose connection, other nodes can still continue to transmit data through the relay, continuity and integrity of the data are ensured, transmission safety between the sensor nodes and the gateway can be ensured by adopting an AES encryption algorithm, a user-defined encryption protocol can be overlaid, and dual data protection is achieved.

The invention discloses a power grid equipment asset data capturing system and method based on multi-platform traceability, and relates to the technical field of data capturing and power grid equipment asset management, and the system comprises a data source analysis module, a dynamic priority calculation module, a task execution and monitoring module, a closed-loop feedback adjustment module, and a system scheduling and interaction control module. According to the method, a control system for analysis, decision making, execution, feedback and adjustment is constructed, data capture is converted from open-loop static execution to closed-loop dynamic optimization, the system can continuously sense multi-dimensional features of structural complexity, updating frequency and value density of a data source, the resource state of the system is monitored in real time, and the real-time performance of the system is improved. Comprehensive decision making is carried out through dynamic priority calculation, the mechanism enables system resources to be preferentially allocated to the highest data source and task in the current environment all the time, and waste of the resources in invalid grabbing, frequent sealing or high-delay tasks is avoided.

The invention relates to a multi-agent-based social media knowledge graph automatic expansion method, and belongs to the technical field of artificial intelligence, natural language processing and knowledge graph construction. The method mainly comprises the following three steps: step 1, a preprocessing stage: a data capture agent legally obtains multi-platform data, a data cleaning agent removes noise, and a feature extraction agent extracts features for different data forms; step 2, an extraction stage: matching a knowledge structure mode by a mode alignment agent, extracting an agent mining entity and a relationship, and processing information contradiction by a conflict resolution agent; and step 3, in a fusion stage, the evaluation agent calculates a knowledge triple score by integrating multiple indexes, and the knowledge graph storage agent is responsible for storage and updating. The social media data can be effectively processed, automatic extension of the knowledge graph is achieved, the quality and coverage range of the knowledge graph are improved, and rich and accurate knowledge support is provided for intelligent application.

The application discloses a kind of general data acquisition system based on RPA robot, comprising;Terminal equipment, server end and background control end, the terminal equipment is connected with the background control end by the server, the server end acquires the terminal equipment in RPA data acquisition state data acquisition system operating state parameter, when RPA is reached in the process of running data running space upper limit, close the application program in suspended state, and whether the remaining space of terminal equipment expected operation meets the normal operation of RPA data extraction is calculated, if not satisfied, then the task amount of RPA data extraction is adjusted, avoid due to RPA data extraction terminal equipment running memory too large, leading to terminal computer equipment system crash.RPA robot data extraction setting task amount is larger in the prior art, computer terminal appears failure under the overload state of RPA robot, and the problem that RPA robot cannot normally run and extract data is solved.

1. The name of the design product: webpage data scraping graphical user interface for electronic devices. 2. The use of the design product: an electronic device. 3. The design points of the design product: in the graphical user interface. 4. The picture or photo that best indicates the design points: front view. 5. The use of the graphical user interface: for accessing applications that allow scraping or capturing webpage data, such as accessing webpage scraping APIs or using artificial intelligence assistants. 6. The human-computer interaction mode of the graphical user interface: it can be interacted through touching or clicking the graphical user interface shown on the display screen of the electronic device or operating the buttons of the electronic device.

The application belongs to the technical field of enterprise informationization and data processing, and particularly relates to a research and development project addition deduction automation processing method and related products, which realizes high-precision quantification and traceability of "person, material and machine" costs through dynamic weight mapping, graph database BOM bloodline atlas and discrete time state retention logic; introduces a clustering algorithm to dynamically generate a compliance threshold conforming to objective laws of the industry, solves the problem of misjudgment of special projects; blocks dirty data through abnormal attribution and interpolation compensation algorithms, ensures the safety and accuracy of tax declaration data; and finally realizes full-process automation from bottom-layer heterogeneous system data grabbing, cache synchronization, compliance determination to final quota calculation, reduces manual intervention cost and compliance risk.

The application discloses a data crawling method and device, computer equipment and a storage medium. The method receives crawling configuration information, the crawling configuration information including a URL of a target crawling page and target crawling content; the target crawling content is parsed to generate a plurality of crawling tasks corresponding to independent data fields; based on the URL of the target crawling page loading a page, data crawling is performed on each crawling task to obtain DOM data and response data of a network request; the DOM data and the response data of the network request are parsed to obtain structured crawling data of the crawling task. The efficiency of data crawling can be improved.

The present disclosure provides a data crawling method and device, relates to the technical field of network application, and can solve the problems of low efficiency and easy errors of the current data crawling method. The specific technical scheme is: a crawler template is configured, and website information to be accessed is obtained; a crawler module is used to set the database structure of a target database; a corresponding target website page is loaded according to the website information to be accessed; a selection operation of a user on the target website page is obtained, and a target position is determined according to the selection operation; a document structure of corresponding content in the target website page is obtained according to the target position, and an XPath expression is generated according to the document structure; a crawler program is called, and target data of the target website page crawled by the crawler program according to the XPath expression is stored in the target database. The present application is used for data crawling.

The application relates to the field of crawlers, in particular to a distributed crawler method and system, which improves the data crawling efficiency, data crawling scheduling capability and flexibility. The scheme comprises the following steps: collecting URL data, saving the crawled URL data into a URL data set, performing hash calculation on the collected URL, if the calculation result does not belong to the URL data set, adding the URL to a task node, otherwise discarding the URL; adopting a neural network to distill the URL, intercepting the URL exceeding a preset length, taking the distilled result as specific features; establishing a DQN model, inputting the specific features into the DQN model, dynamically distributing the crawling tasks according to the sub-node throughput, and performing data grabbing; and mining preference data in the data grabbing process through a knowledge graph. The application is suitable for data crawling.

The invention relates to the technical field of cross-border financial transaction security and block chain application, and discloses a block chain-based cross-border financial transaction risk real-time prevention and control method, which comprises the following operation steps: S1, establishing a multi-chain composite distributed account book architecture, deploying a PBFT + PoA hybrid consensus mechanism, constructing a transaction main chain, a privacy data channel and a supervision side chain, and constructing a block chain; basic operation of transaction metadata storage, sensitive data encryption transmission and supervision data capture channels is guaranteed; s2, establishing a cross-chain identity hub and data fusion system; according to the cross-border financial transaction risk real-time prevention and control method and system based on the block chain, a multi-chain composite distributed account book architecture is established, a PBFT + PoA hybrid consensus mechanism is deployed, and a transaction main chain, a privacy data channel and a supervision side chain are separated, so that the traceability of transaction metadata is ensured, encrypted transmission of sensitive data is realized, and the risk of the cross-border financial transaction is prevented and controlled in real time. And meanwhile, an exclusive data capturing channel is provided for a supervision mechanism.

The invention provides a multi-dimensional index scoring method and system based on a rule engine. The method comprises the following steps: constructing an index set according to user configuration operation; defining a metadata parent class corresponding to the index set and a handler subclass corresponding to each index; traversing each handler subclass by adopting a calculation engine, and executing the grasp logic of each handler subclass in batches to obtain index set metadata; performing metadata data structure analysis on the index set metadata to generate a metadata description file associated with each index; writing a scoring calculation rule of each index according to each metadata description file and a preset calculation structure, and calling the scoring calculation rule of each index by adopting a rule engine to obtain scoring data; and carrying out multi-dimensional index aggregation processing on the scoring data to obtain a target scoring report, and carrying out visual display on the target scoring report in an index billboard. According to the solution, the access period of the newly added indexes can be effectively reduced, and the expansion capability of the system is improved.

Exemplary embodiments of the present disclosure are directed towards a system for scraping product prices, providing alerts for price changes and inventory status. The system includes a computing device with a user interacting module, allowing users to select products by entering names and preferences. The module transmits product details to a server over a network. The server comprises a data scraping module that receives the details and utilizes Selenium API and website API to collect and compare prices from multiple online sources, identifying the lowest price. The system generates alerts for the lowest price and inventory status, which are sent back to the computing device. The user interacting module allows users to view these alerts, accept or reject deals, and facilitates order placement and payment processing for accepted deals. If a deal is rejected, the system repeats the price comparison process and issues low inventory alerts if necessary.

The invention belongs to the technical field of insurance data processing, and particularly relates to a bank-insurance data docking method and system. According to the bank-insurance data docking method, standardized definition is carried out on fields, with the same meaning but different names, of a bank and an insurance through a standard field information base, automatic recognition of a matching module is facilitated, and matching and docking of data are completed. The automatic capturing module and the matching module replace manual export, comparison and table modification operations, so that the data processing time is greatly shortened, and the efficiency is improved. For personalized fields, manual intervention can be carried out through the alarm module, secondary recognition is carried out manually, and finally matching of full-quantity fields is achieved. The log module can completely record the whole process operation information from data capture to message generation, a clear and traceable data link is provided for management and troubleshooting, and the subsequent operation and maintenance difficulty is reduced.

The invention provides a dual-system performance monitoring method and system based on dynamic configuration and multi-tool linkage. The dual-system performance monitoring method comprises the following steps: dynamically setting a performance target threshold value for a monitored object according to a use requirement; wherein a plurality of monitoring objects are provided; establishing a corresponding relation between the monitoring object and at least one performance analysis tool, and configuring a data processing strategy; continuously monitoring the performance index of the monitored object; when it is monitored that the performance index exceeds a performance target threshold value, the performance analysis tool is controlled to execute a data processing strategy; the data processing strategy comprises data capture and analysis of a monitored object. The technical problems that in the related technology, log records of all modules are not detailed, so that log output content and video content are inconsistent, it is difficult to accurately correspond to the actual running state of a system during performance analysis, reasons for high loads of all the modules cannot be rapidly analyzed, repeated testing of testers is needed, and the efficiency is high are solved. And the time cost of development and analysis is increased.