75 results about "Data scraping" patented technology

The invention discloses an intelligent public opinion research and judgment method and system for public security actual combat, and the method achieves the high-throughput data capture and real-time public opinion inflow through multi-source crawler, breakpoint resume, incremental collection and asynchronous concurrent processing, and greatly reduces the workload of manual screening. A unified, quantifiable and structured public opinion risk grading standard is established, and risk grading standard unification is realized by constructing three types of structured public opinion dictionaries of event subjects, event properties and sensitivity grades; the grading rule is quantified and objective and adapts to the grading system of the public security actual combat, and the subjectivity and inconsistency of research and judgment are remarkably reduced; and the recessive public opinion recognition capability is improved through the deep semantic model. According to the method, the BERT model, keyword vector fusion and the police affair large model are combined, so that accurate analysis of complex semantics such as metaphor, black talk and irony mood is realized; high-robustness semantic matching and duplicate removal are carried out; through accurate extraction of risk factors and event identification, the problem of high missed judgment rate in the prior art is improved.

The invention provides a manipulator commodity grabbing system and method based on visual recognition, and relates to the technical field of equipment control, and the method comprises the steps: collecting the image information of a commodity in a working area based on a multi-modal visual collection device, analyzing the commodity image information, and obtaining the depth information of a commodity image; the intelligent detection module is used for intelligently analyzing the depth information of the commodity image, extracting image features of the commodity image and determining attribute information of the commodity according to the image features; the optimal grabbing box parameter acquisition module is used for analyzing the attribute information of the commodity to generate optimal grabbing box parameters; the spatial pose acquisition module is used for determining spatial pose data of the manipulator during commodity grabbing according to the optimal grabbing frame parameters; and the grabbing control module is used for generating a grabbing control instruction of the manipulator according to the spatial pose data, and controlling the manipulator to grab the commodity according to the grabbing control instruction. And the grabbing success rate and the operation efficiency in a complex scene are remarkably improved.

The invention belongs to the technical field of data safety, and particularly relates to a visual safety brain modeling and early warning platform of an expressway toll collection system, which comprises a data capturing module used for acquiring real-time interaction data and a toll station three-dimensional model of the expressway toll collection system, and constructing an expressway toll station panoramic three-dimensional model in a three-dimensional environment; the dynamic display module comprises a first floating information board, a second floating information board, a third floating information board, a fourth floating information board and a fifth floating information board; the early warning module is used for analyzing the real-time interaction data acquired by the data capturing module to find abnormal data and tracing an abnormal event; and the adjusting module provides a dynamic adjusting mechanism according to the matching result. According to the method, the distribution strategy is adjusted in real time to cope with changes of emergency situations and tasks, the empty window period occurring in the adjustment process is avoided as much as possible, and network security management vulnerabilities of the highway toll collection system can be prevented.

The application provides a mechanical hand commodity grabbing system and method based on visual recognition, and relates to the technical field of equipment control. The system comprises the following steps: collecting image information of commodities in a working area based on a multi-modal visual collection device, analyzing the commodity image information, and obtaining depth information of the commodity image; an intelligent detection module is used for intelligently analyzing the depth information of the commodity image, extracting image features of the commodity image, and determining attribute information of the commodity according to the image features; an optimal grabbing frame parameter acquisition module is used for analyzing the attribute information of the commodity to generate optimal grabbing frame parameters; a spatial pose acquisition module is used for determining spatial pose data of the mechanical hand in grabbing the commodity according to the optimal grabbing frame parameters; and a grabbing control module is used for generating grabbing control instructions of the mechanical hand according to the spatial pose data, and controlling the mechanical hand to grab the commodity according to the grabbing control instructions. The grabbing success rate and work efficiency in a complex scene are significantly improved.

The invention provides an image data processing method and device, equipment, a storage medium and a program product, and the method comprises the steps: responding to a first display interrupt signal arriving according to a preset period and a data capture request, determining a state of a coding operation of first image data triggered based on a second display interrupt signal arriving according to a preset period, the second display interrupt signal arriving earlier than the first display interrupt signal, and the first image data being image data of a first frame stored in a frame buffer queue before the second display interrupt signal arrives; triggering a third display interrupt signal in response to determining that the encoding operation of the state representation for the first image data is completed; and based on the third display interrupt signal, obtaining second image data currently stored in the first frame from the frame buffer queue, and processing the second image data. According to the method, the display interruption can be dynamically adjusted, so that the coding efficiency can be improved, and the frame loss ratio can be reduced.

The application provides an automobile IBC product offline appearance identification detection system and method, and relates to the technical field of automobile part detection. The system comprises: a feeding vision module for identifying the placement posture of the IBC product to be detected on the conveying line and outputting posture data; a mechanical hand module for grabbing the product according to the posture data and transferring it to a detection station; a rotary table module arranged at the detection station for fixing and rotating the product; a vision detection module for scanning the surface of the product during rotation, acquiring three-dimensional image data, and performing defect detection and classification based on the three-dimensional image data; a data tracing module in communication with a central control system and an enterprise MES system for binding the unique identification of the product and the detection data; and the central control system for coordinating the operation of each module and controlling the mechanical hand module to perform a sorting and stacking operation corresponding to the result according to the result of defect detection and classification. Full automation and full coverage of the detection process are realized.

The invention discloses a perimeter intrusion monitoring method and system for a tunnel portal high-speed rail line, and relates to the technical field of intelligent monitoring, and the method comprises the steps: obtaining a line tunnel portal sequence driving data set according to a high-speed rail line advancing tunnel portal sequence of a sorted target high-speed rail line, and carrying out the intrusion risk recognition, obtaining a tunnel portal sequence intrusion risk coefficient set; meanwhile, interference analysis is carried out on a high-speed rail line advancing tunnel portal sequence, a tunnel portal sequence interference factor set is obtained, a tunnel portal sequence intrusion risk coefficient set is corrected, and a preset video data capture frequency is determined based on the corrected tunnel portal sequence intrusion risk correction coefficient set; and then differentiated monitoring analysis is carried out on the high-speed rail line tunnel portal, and tunnel portal perimeter intrusion information is determined for early warning management and control. The technical effects that differentiated intelligent monitoring and analysis of the tunnel portal are achieved, the accuracy of intelligent monitoring and analysis and the processing efficiency of intrusion monitoring data are improved, and then the timeliness of early warning of the perimeter intrusion of the tunnel portal is ensured are achieved.

The invention discloses a multi-protocol data capturing method and system, and relates to the technical field of data capturing. The multi-protocol data capturing method is applied to the multi-protocol data capturing system, and the multi-protocol data capturing system comprises a client and a server which is in communication connection with the client through multiple network protocols. The multi-protocol data capturing method comprises the following steps: a client sends a data capturing request to a server through one of a plurality of network protocols; the server calls a corresponding processor to perform data capture based on the request type carried in the data capture request; the server converts the grabbed data into a unified intermediate format; and the server converts the data in the unified intermediate format to the client through one of the plurality of network protocols and sends the data to the client. According to the multi-protocol data capturing method and system disclosed by the invention, cross-language and cross-protocol data capturing service can be realized.

The invention discloses an automatic processing system and method for electric power information communication scheduling operation data. The system comprises a man-machine interaction module, a data capture module, a report processing module, a file management module and a data verification module. The man-machine interaction module is used for realizing data interaction between the system and an operator; the data capturing module is used for automatically extracting work order data through a cross-communication scheduling core business system; the report processing module is used for realizing monthly report updating, daily report generation and end-of-month data archiving; the file management module is used for managing template files, result files and log files related to the system; and the data verification module is used for verifying the accuracy of the data processing result. The method disclosed by the invention is high in data processing efficiency, good in integrity and relatively high in accuracy.

The application discloses a kind of methods for assisting large model networking query, the method is by receiving and encapsulating as standardization demand to parse user query request, matching dynamic residential proxy IP pool, IP dynamic rotation and load balancing mechanism are started, generate standardization collection task, realize multi-platform data scraping through multi-tool cooperation, anti-crawling avoidance and high concurrency scheduling, clean, double check and structure transformation are carried out to original data, add standardization field annotation, push structured data to large model and monitor data satisfaction degree;The application is through dynamic residential proxy IP pool and IP rotation mechanism, avoid regional restriction and anti-crawling interception, with headless browser, API identification technology, adapt to dynamic and waterfall page, improve collection integrity and efficiency, after cleaning, check and structure transformation, combined with automatic completion query condition and daily optimization mechanism, the accuracy, timeliness and stability of large model real-time query response are greatly improved.

This application discloses a method, apparatus, device, and storage medium for constructing a dataset based on 3D games. The method first obtains a compiled game data scraping script for the target 3D game; runs the target 3D game and obtains game screen data based on the game data scraping script, wherein the game screen data includes image data and non-image data; then cleans the game screen data to obtain sample data to be labeled; finally, a data labeling system is constructed, and the sample data is labeled using the data labeling system. It can be seen that this application achieves the acquisition of synthetic data from 3D games, which is more efficient than data acquisition in the real world. Furthermore, the synthetic game data can extract corresponding semantic information from the game, which can greatly simplify the data labeling process, enabling data obtained from games to be used for model training.

The application provides a BOP data acquisition EWI generation method, medium and system, and belongs to the technical field of production business data.The application firstly constructs a multidimensional BOP data load matrix, accurately calculates the contribution value of each parameter through a thermodynamic entropy equation, then establishes a standard EWI basic matrix, and optimizes the BOP data acquisition strategy and calculates the data integrity score by using the minimum spanning tree algorithm.In the data processing stage, the EWI mergable index is accurately generated by using the Lagrange multiplier method, the transverse error is calculated by using the Euclidean distance function to analyze the spatial relationship between parameters, the longitudinal error is calculated by using the Fourier transform to monitor the time change trend of parameters, the two kinds of errors are comprehensively formed into a hybrid error evaluation result, and finally, the comprehensive index accurately reflecting the production quality state is output in combination with the EWI mergable index.

The invention relates to an analysis method and device for a chip. The method comprises the steps that data capture is conducted on latest results of a target chip by different analysis tools to obtain target data, the number of the analysis tools is at least two, and analysis conducted on the target chip by the analysis tools is different; taking the target data corresponding to each analysis tool as an analysis basis and sending the analysis basis to other analysis tools; performing anomaly evaluation based on the target data to obtain an evaluation result; under the condition that the evaluation result is abnormal, the abnormal evaluation result serves as an analysis basis for optimization and is sent to a first tool in the analysis tools, and all the analysis tools are controlled to end analysis until an end condition is met; the end condition is that the evaluation result is normal. The accuracy and speed of chip analysis are remarkably improved, multi-dimensional linkage analysis is achieved, and the method is suitable for multiple research and development stages of chip development and wide in application range.

The invention discloses an abstract content generation method and device based on improved Bert, and the method comprises the steps: firstly setting a seed URL, selecting a webpage link, capturing webpage data through employing a crawler frame, carrying out the duplication removal of the webpage data, carrying out the classified storage of webpage contents through employing an efficient storage mechanism, and carrying out the generation of abstract contents based on the classified storage of the webpage contents. The method comprises the following steps of: performing multi-level filtering and identification on advertisement contents by using an HTML (Hypertext Markup Language) structure, URL (Uniform Resource Locator) and href identification and a neural network model, outputting an advertisement dichotomy result, performing image-text preprocessing by using an improved Bert model based on a dynamic self-adaptive decision boundary, and generating a webpage abstract by combining semantic information of texts and images through a theme-enhanced multi-modal fusion method. According to the method, data capture and storage are carried out in combination with a crawler framework, advertisement multi-level filtering is carried out based on rules and a neural network model, and a dynamic self-adaptive decision boundary image-text preprocessing and theme enhancement multi-modal fusion method of an improved Bert model is adopted, so that efficient webpage data automatic processing is realized.

The invention discloses a mechanical arm grabbing anti-sliding control method, and relates to the technical field of intelligent sorting and recycling, and the mechanical arm grabbing anti-sliding control method comprises the following steps: collecting visual information of an object in real time, and determining grabbing information of a clamping jaw on a mechanical arm based on the visual information; wherein the visual information comprises displacement data, three-dimensional shape data and anti-skid data, and the grabbing information comprises a grabbing position, grabbing force, a clamping jaw opening and closing angle and a mechanical arm tail end attitude angle. Object characteristics are accurately described through displacement data, three-dimensional shape data and anti-slip data, adaptive grabbing information is output in combination with a machine learning model, and the slip risk is reduced from the source; a secondary grabbing mechanism with the preset sliding path matched with the real-time sliding path is designed according to the sliding risk, the multiple preset sliding paths are dynamically simulated, the optimal preset sliding path is dynamically matched in combination with the real-time track, rescue grabbing of the sliding object can be achieved, and the grabbing success rate can be increased.

This application discloses an adaptive data crawling method, device, storage medium, and apparatus, relating to the field of Internet technology. The method includes: configuring request parameters based on an interface request initiated by a target user, a preset configuration file, and a preset custom mapping rule table to obtain data configuration information; performing pre-crawling verification on the data configuration information to obtain pre-verification results; and matching a target-level thread pool from a preset-level thread pool based on the pre-verification results and a preset sliding window algorithm, and activating threads in the target-level thread pool to complete data crawling. Compared to traditional data crawling methods that require manual intervention for system updates and maintenance, affecting the accuracy and success rate of data crawling, this application can automatically map request parameters, notify the custom mapping rule table to assemble complex parameters and perform pre-verification of parameters, and filter thread priorities to complete data crawling, thereby improving the accuracy and success rate of data crawling.

This invention relates to a method, apparatus, equipment, and medium for automatic configuration and cost calculation of miscellaneous charges. The method includes constructing a multi-dimensional standardized template library and completing import verification to form a set of callable pricing rules; upon receiving a calculation request, retrieving and preprocessing order-related data across systems; filtering candidate rules through a rule matching engine and determining the optimal billing basis by weighting and sorting according to a preset priority system; adapting the billing mode and formula based on this basis, initiating batch parallel calculations and handling anomalies synchronously; and supporting multi-format export and cross-system synchronization after integrating the results. This invention includes modules for template library construction and verification, data retrieval and preprocessing, rule matching and sorting, cost calculation and anomaly handling, and result integration and synchronization. It achieves automatic configuration and efficient calculation of miscellaneous charges, improves data accuracy and process closure, adapts to the needs of multiple scenarios throughout the warehousing and logistics process, and significantly reduces labor costs.

The invention relates to the technical field of data detection, in particular to a real-time anomaly detection method based on multi-modal data fusion, and the method comprises the following steps: S1, constructing a data capturing platform, obtaining data information of different data types through the data capturing platform, preprocessing the data information, and generating modeling data corresponding to the different data types; s2, performing multi-scale feature extraction on the generated multi-modal modeling data to obtain three-dimensional point cloud data and data features, dynamically calculating a mutual relation between the three-dimensional point cloud data and the data features based on a cross-modal collaborative attention mechanism, and performing dynamic feature reconstruction on the three-dimensional point cloud data and the data features to obtain three-dimensional point cloud data; and performing multi-modal fusion on the reconstructed three-dimensional point cloud data and the data features to obtain final fusion features, thereby effectively solving the problems that an existing real-time anomaly detection method can only perform detection according to single data, the detection accuracy is low, an anomaly root cannot be quickly positioned during detection, and the working efficiency is low.

The invention relates to the technical field of data capture, in particular to a platform development data capture method and system based on SCADA (supervisory control and data acquisition). Comprising the following steps: S1, capturing historical data and real-time data by using an SCADA system, obtaining a data storage module corresponding to the historical data, connecting a cloud platform, and transferring the real-time data to the cloud platform for distribution; real-time frequency and timing frequency data are distinguished, effective time is set for timing data, report deadline is determined based on the shortest remaining effective time, it is avoided that expired data are used to affect an analysis result, and then missing data types needed for generating a complete report are actively recognized, so that the report deadline accuracy is improved. And before the target capture time, the cloud platform instructs the SCADA system to directionally capture the missing data, so that data completion and report generation are ensured to be completed before the deadline, and the problem that the report is lagged or incomplete due to incomplete data acquisition and aging failure in an industrial scene is solved.

The embodiment of the invention discloses a data capturing method and device, a medium and a product. The method comprises the steps that an interface combination and exclusive data blocks of all interfaces in the interface combination are determined, the interface combination comprises at least two interfaces, the at least two interfaces are complementary on the data block downloading speed level, and an exclusive data block combination corresponding to the interface combination comprises all data blocks of a target file; corresponding data blocks in a target file are downloaded through the interfaces in the interface combination, and the identifier of each data block carries a file identifier; and for each file identifier, when it is detected that all the data blocks under the file identifier are downloaded, determining a target file for the file identifier according to all the data blocks under the file identifier. According to the technical scheme provided by the embodiment of the invention, the file downloading speed can be improved under the condition that the interface transmission bandwidth is not changed.

This invention discloses a fault identification and alarm system and method for silicon steel furnace rolls based on the gradient boosting tree method. Based on the gradient boosting tree method, it utilizes furnace roll operation data within the silicon steel furnace, including time-series data and dimensional space data on rotational speed and current, to accurately identify, classify, and alarm for faults such as slight bending, heavy bending, and carbon sleeve wear of the furnace rolls. The steps include capturing furnace roll data from the existing IBA system, analyzing and filtering data in the frequency domain using dimensionality algorithms and existing AI models, and identifying and alarming the operating conditions. This invention extracts data by adding a developed plug-in to the IBA PDA, and through analysis, statistical calculation, and fault analysis of this data, it provides a direct understanding of the furnace roll's performance, technical level, and potential improvements, and triggers pre-defined alarm events to alert workers.

This invention discloses an automatic extraction method for subway train passing data based on time-frequency features, relating to the field of rail transit monitoring. The method includes: collecting subway track vibration signals and performing wavelet transform to obtain a two-dimensional time-frequency map; constructing a wheelset recognition plugin based on a YOLO model to recognize the two-dimensional time-frequency map and obtain initial wheelset recognition results; obtaining a timestamp interval sequence, calculating the coefficient of variation of the timestamp interval sequence, filtering initial wheelset recognition results with a coefficient of variation greater than or equal to a threshold to obtain a negative sample set, and using the remaining initial wheelset recognition results as a multi-train sample set; performing feature extraction based on a random sample consensus algorithm to obtain a positive sample set; performing transfer learning on the wheelset recognition plugin to obtain an optimized wheelset recognition plugin, performing wheelset recognition, and calculating and obtaining subway train passing data. This method solves the problems of inconsistent data acquisition standards, non-standard processing procedures, low overall efficiency, and insufficient reliability in existing technologies.

The invention discloses a distributed garden ecological data acquisition system based on LoRa, and belongs to the technical field of data acquisition. The distributed garden ecological data acquisition system based on LoRa comprises a data acquisition module, a data transmission module, a data processing module and a data storage module. The problems that in the prior art, data capture is incomplete or interrupted possibly, manual collection efficiency is low, and large-scale data collection requirements are difficult to meet are solved, data collection is carried out through the LoRa terminal nodes, data transmission levels are reduced, data collection and transmission efficiency is improved, the overall reliability of the system is improved, and the system is suitable for popularization and application. Even if some nodes temporarily lose connection, other nodes can still continue to transmit data through the relay, continuity and integrity of the data are ensured, transmission safety between the sensor nodes and the gateway can be ensured by adopting an AES encryption algorithm, a user-defined encryption protocol can be overlaid, and dual data protection is achieved.

The invention discloses a real-time analysis system based on business data isolation and a low-intrusive data synchronization method, and belongs to the technical field of big data analysis and data security, and the system comprises a business data layer which is used for storing original business data and comprises a business database cluster and a business system; the synchronous agent module is deployed between the business data layer and the analysis data layer, realizes non-intrusive data capture by analyzing a database incremental log, and performs field-level encryption and fragmentation transmission on data; and the analysis data layer is used for storing the desensitized analysis data by adopting a column type storage architecture, and realizing real-time query and authority control of the encrypted data through a unified query gateway and a dynamic desensitization module. According to the method, safe, efficient and real-time data isolation analysis can be realized through innovative technical combination.

The invention provides a medical business opportunity management system, method and device based on an AI agent and a medium, and belongs to the technical field of information processing. The system comprises a business opportunity source information maintenance module which is used for classifying and identifying Internet information sources through a large language model, establishing an information source database, periodically scanning and updating bidding announcement links through an intelligent search agent, and auditing the bidding announcement links through an auditing mechanism; the business opportunity data capturing module is used for receiving the approved bidding and tendering announcement link, performing page preprocessing through a JinaReader model, performing deep analysis by utilizing an AI model, extracting key information, performing data verification through a rule engine, generating structured data and storing the structured data in a database; and the business opportunity data statistical query module is used for providing a user interface, supporting multi-dimensional query condition input, retrieving business opportunity data from the database, performing statistical analysis and displaying a query result.

The invention discloses a power grid equipment asset data capturing system and method based on multi-platform traceability, and relates to the technical field of data capturing and power grid equipment asset management, and the system comprises a data source analysis module, a dynamic priority calculation module, a task execution and monitoring module, a closed-loop feedback adjustment module, and a system scheduling and interaction control module. According to the method, a control system for analysis, decision making, execution, feedback and adjustment is constructed, data capture is converted from open-loop static execution to closed-loop dynamic optimization, the system can continuously sense multi-dimensional features of structural complexity, updating frequency and value density of a data source, the resource state of the system is monitored in real time, and the real-time performance of the system is improved. Comprehensive decision making is carried out through dynamic priority calculation, the mechanism enables system resources to be preferentially allocated to the highest data source and task in the current environment all the time, and waste of the resources in invalid grabbing, frequent sealing or high-delay tasks is avoided.

The invention discloses an SCADA (supervisory control and data acquisition)-based conjoined panel shared power supply control system data capturing method, which belongs to the technical field of data capturing and specifically comprises the following steps of: acquiring three types of heterogeneous data, namely power supply analog waveform, panel digital state and system event log through a dedicated channel of an SCADA and conjoined panel shared power supply control system; then time consistency fusion is carried out, a unified high-precision time base is established, data are mapped, and transmission interruption data segments are reconstructed; performing multi-dimensional feature perception on the fused data, identifying a waveform mode, analyzing a state transition path, and mining an event trigger chain; dynamically constructing a data capture rule set containing three types of constraints based on a sensing result, compiling the data capture rule set into an executable protocol, loading the executable protocol to an SCADA data stream processing engine, and judging a real-time data stream on line by the engine and extracting and outputting a structured data fragment sequence; according to the invention, high-quality data support is provided for accurate monitoring and operation analysis of the connected panel shared power supply control system.

The invention relates to a multi-agent-based social media knowledge graph automatic expansion method, and belongs to the technical field of artificial intelligence, natural language processing and knowledge graph construction. The method mainly comprises the following three steps: step 1, a preprocessing stage: a data capture agent legally obtains multi-platform data, a data cleaning agent removes noise, and a feature extraction agent extracts features for different data forms; step 2, an extraction stage: matching a knowledge structure mode by a mode alignment agent, extracting an agent mining entity and a relationship, and processing information contradiction by a conflict resolution agent; and step 3, in a fusion stage, the evaluation agent calculates a knowledge triple score by integrating multiple indexes, and the knowledge graph storage agent is responsible for storage and updating. The social media data can be effectively processed, automatic extension of the knowledge graph is achieved, the quality and coverage range of the knowledge graph are improved, and rich and accurate knowledge support is provided for intelligent application.

The application discloses a kind of general data acquisition system based on RPA robot, comprising;Terminal equipment, server end and background control end, the terminal equipment is connected with the background control end by the server, the server end acquires the terminal equipment in RPA data acquisition state data acquisition system operating state parameter, when RPA is reached in the process of running data running space upper limit, close the application program in suspended state, and whether the remaining space of terminal equipment expected operation meets the normal operation of RPA data extraction is calculated, if not satisfied, then the task amount of RPA data extraction is adjusted, avoid due to RPA data extraction terminal equipment running memory too large, leading to terminal computer equipment system crash.RPA robot data extraction setting task amount is larger in the prior art, computer terminal appears failure under the overload state of RPA robot, and the problem that RPA robot cannot normally run and extract data is solved.

The application provides a PCIe-based data transmission method, device and system. The method is executed at a device end. A plurality of ports of the device end are identified by a driver of a host end to match corresponding data buffers. The method comprises: segmenting a data packet into data packets; generating a descriptor by a descriptor generator according to the data packets; polling a channel corresponding to each port by the descriptor to detect the descriptor; transmitting the detected descriptor to be processed to a DMA controller to generate control information and send the control information to a data grabber and a data transmitter; reading the data packets from the corresponding channel by the data grabber and transmitting the data packets to the data transmitter; and transmitting the data packets to the data buffers of the host end by the data transmitter to process the data by the driver. The application can simplify the transmission process and reduce the data transmission delay.