142 results about "Data scraping" patented technology

The invention discloses a website AI intelligent risk assessment real-time evasion system and an evasion method thereof. The system comprises a VPN connection module to ensure that data capture is safe and hidden; the data capturing module collects various types of information of a target website in real time; the risk feature extraction module performs multi-dimensional feature extraction; the AI risk analysis and evaluation model identifies risks from multiple aspects of images, scripts and the like and grades the risks; the risk avoiding and warning module executes a hierarchical response strategy according to a result; the feedback and model optimization module collects a feedback and sample optimization model. According to the method, website risk avoidance is realized through the steps of VPN connection, data acquisition, feature extraction, risk assessment, strategy execution, model optimization and the like. According to the method, VPN and artificial intelligence technologies are combined, website risks can be comprehensively evaluated in real time, access strategies are dynamically adjusted, complex network attacks are effectively coped, and user network access safety is guaranteed.

The invention discloses an anti-phishing webpage detection system and method. The anti-phishing webpage detection system comprises a webpage data capture module, a risk analysis module, a malicious webpage judgment module and a response decision module, relates to the technical field of anti-phishing webpage detection. According to the method, multi-dimensional data, including URL information, domain name information and webpage content information, of a webpage are extracted through the webpage data capturing module, a comprehensive original data basis is provided, and accurate support is provided for follow-up risk analysis; according to the method, comprehensive risk analysis is carried out by utilizing various webpage features including URL features, domain name features and content features, and potential phishing webpages can be effectively identified by calculating risk scores of the URL, the domain name and the webpage content; according to the method, the web pages are classified according to the comprehensive risk scores, the high-risk web pages are intercepted, the emergency response logs are generated, the low-risk web pages are continuously monitored, and safe browsing of a user is ensured.

The invention discloses a general news page data acquisition method, device and equipment, a medium and a product, and relates to the technical field of news page data acquisition. The method comprises the following steps: sending an HTTP (Hyper Text Transport Protocol) request to a target news website to obtain return data of a news webpage, calling a browser automation tool Selenium to perform data capture after all elements of the news webpage are loaded when judging that dynamic loading contents exist, and taking a capture result as original data of the news webpage, then, for each node in the DOM tree, calculating to obtain corresponding text density and symbol density, judging whether the webpage content is complex content or not based on a calculation result, and if the webpage content is complex content, analyzing the webpage content through multi-dimensional feature analysis and a big language model used for news page analysis based on rules; and positioning to obtain a final extraction result aiming at the news page key information, and finally carrying out data cleaning and standardization processing on the extraction result to obtain news page data with a uniform format and outputting the news page data.

The invention provides a grabbing control method and device of a robot, the robot, a storage medium and a program product. A first collecting assembly is controlled to collect first sensing data in the current environment where the robot is located, and a mechanical arm is controlled to drive a second collecting assembly to collect second sensing data in the current environment; and controlling the mechanical arm to grab the target object based on the first sensing data and the second sensing data. Thus, the target object can be grabbed according to the first sensing data and the second sensing data obtained by the first collection assembly and the second collection assembly which are different in position, the target object can be grabbed according to data of different view angles, the possibility of misrecognition caused by limited obtained information under a single view angle is reduced, and the recognition accuracy is improved. And the accuracy and the flexibility of the grabbing control method are improved.

The invention discloses an intelligent public opinion research and judgment method and system for public security actual combat, and the method achieves the high-throughput data capture and real-time public opinion inflow through multi-source crawler, breakpoint resume, incremental collection and asynchronous concurrent processing, and greatly reduces the workload of manual screening. A unified, quantifiable and structured public opinion risk grading standard is established, and risk grading standard unification is realized by constructing three types of structured public opinion dictionaries of event subjects, event properties and sensitivity grades; the grading rule is quantified and objective and adapts to the grading system of the public security actual combat, and the subjectivity and inconsistency of research and judgment are remarkably reduced; and the recessive public opinion recognition capability is improved through the deep semantic model. According to the method, the BERT model, keyword vector fusion and the police affair large model are combined, so that accurate analysis of complex semantics such as metaphor, black talk and irony mood is realized; high-robustness semantic matching and duplicate removal are carried out; through accurate extraction of risk factors and event identification, the problem of high missed judgment rate in the prior art is improved.

The invention discloses a manipulator target grabbing path intelligent planning method based on visual depth perception. The manipulator target grabbing path intelligent planning method comprises the following steps that grabbing target point cloud data are obtained and preprocessed to generate a structured point cloud data set; executing SuperPoint division and adjacency relation construction, and generating a SuperPoint graph structure; constructing a plurality of groups of action guide graphs and embedding the action guide graphs into an SPGNet attention mechanism to form an improved SPGNet structure; inputting structured point cloud data, extracting node features and capturing response, and generating a capturing candidate region and a sorting result; determining grabbing point coordinates and grabbing posture parameters according to the sorting result; path search and obstacle prediction are executed in combination with the tail end pose of the mechanical arm, and grabbing path trajectory data are generated; new point clouds are collected in the grabbing process, a structure topology self-adaptive updating unit is called to reconstruct a graph structure, and a final control instruction is output. According to the invention, high-precision grabbing and path self-adaptive control of the manipulator in a complex environment are realized, and the dynamic operation stability is effectively improved.

The invention discloses a depression detection method based on image-text social media analysis, relates to the technical field of automatic detection of depression, and solves the technical problem in the prior art that whether a user is a depression patient or not is detected by analyzing character and picture data published by the user on a social media platform. Specifically, the method comprises four steps of data capture, post screening, image-text representation and fusion classification. Generally speaking, the method comprises the following steps: firstly, capturing a posted document or posted post of a user in a certain time period from a social media platform, including characters and picture data in the posted document; then, performing depression related symptom prediction on the posted document, and screening out the most related K posts; then, respectively carrying out text and picture representation on the screened posts; and finally, fusing the characterization vectors of the two modes for depression detection.

The invention discloses a large model algorithm capability data set construction method and device and electronic equipment, and belongs to the technical field of large models.The method comprises the steps that data capture is conducted from multiple platforms through a data collection module, and original data are obtained; performing data cleaning on the original data by using a data cleaning module to obtain target cleaned data; performing correctness verification on the target cleaning data by using a correctness verification module to obtain verification data; performing algorithm labeling on the verification data by using an algorithm labeling module to obtain labeled data; performing data standardization on the annotated data by using a data standardization module to obtain standardized data; and evaluating the standardized data by using the evaluation framework module to obtain an evaluation result. According to the method, wide data coverage and high quality are ensured; advantages and disadvantages of the model in different algorithm types can be deeply analyzed; the evaluation result is more accurate; an interactive tool is provided, and the use cost of a user is greatly reduced.

The invention provides a manipulator commodity grabbing system and method based on visual recognition, and relates to the technical field of equipment control, and the method comprises the steps: collecting the image information of a commodity in a working area based on a multi-modal visual collection device, analyzing the commodity image information, and obtaining the depth information of a commodity image; the intelligent detection module is used for intelligently analyzing the depth information of the commodity image, extracting image features of the commodity image and determining attribute information of the commodity according to the image features; the optimal grabbing box parameter acquisition module is used for analyzing the attribute information of the commodity to generate optimal grabbing box parameters; the spatial pose acquisition module is used for determining spatial pose data of the manipulator during commodity grabbing according to the optimal grabbing frame parameters; and the grabbing control module is used for generating a grabbing control instruction of the manipulator according to the spatial pose data, and controlling the manipulator to grab the commodity according to the grabbing control instruction. And the grabbing success rate and the operation efficiency in a complex scene are remarkably improved.

A method of extracting and conditioning content for use in a hierarchically structured merchant risk monitoring environment may include receiving an indication of a merchant under consideration, identifying one or more web pages likely to have information associated with travel exposure for the merchant under consideration based on the indication, conducting data scraping on the identified one or more web pages to attempt to obtain scraped travel exposure data, responsive to obtaining the scraped travel exposure data, employing a machine learning module to determine a travel risk exposure estimate based on the scraped travel exposure data, and responsive to not obtaining the scraped travel exposure data, determining the risk exposure estimate based on a predefined travel risk estimate associated with an industry classification of the merchant under consideration.

The invention belongs to the technical field of data acquisition, and discloses a method for realizing data capture in combination with a browser plug-in, the scheme is based on Chrome plug-in extension and realizes target website data capture through a mode of simply configuring an address of a website needing to be captured and a data interface URL (Uniform Resource Locator), and the scheme comprises the following specific steps: step 1, task plan configuration; step 2, task information configuration; step 3, executing a capturing task; 4, capturing and storing data; and 5, uploading the data. According to the method, the real Chrome browser is started and the Tampermonkey plug-in is combined for crawling, an anti-crawling strategy can be effectively bypassed, the data capturing efficiency is improved, compared with a traditional method, according to the scheme, all data of the server are configured through the configuration file, the target website data can be rapidly collected in a specified mode, operation is easy, and the data capturing efficiency is improved. And the learning cost is reduced.

The invention belongs to the technical field of data safety, and particularly relates to a visual safety brain modeling and early warning platform of an expressway toll collection system, which comprises a data capturing module used for acquiring real-time interaction data and a toll station three-dimensional model of the expressway toll collection system, and constructing an expressway toll station panoramic three-dimensional model in a three-dimensional environment; the dynamic display module comprises a first floating information board, a second floating information board, a third floating information board, a fourth floating information board and a fifth floating information board; the early warning module is used for analyzing the real-time interaction data acquired by the data capturing module to find abnormal data and tracing an abnormal event; and the adjusting module provides a dynamic adjusting mechanism according to the matching result. According to the method, the distribution strategy is adjusted in real time to cope with changes of emergency situations and tasks, the empty window period occurring in the adjustment process is avoided as much as possible, and network security management vulnerabilities of the highway toll collection system can be prevented.

The application provides a mechanical hand commodity grabbing system and method based on visual recognition, and relates to the technical field of equipment control. The system comprises the following steps: collecting image information of commodities in a working area based on a multi-modal visual collection device, analyzing the commodity image information, and obtaining depth information of the commodity image; an intelligent detection module is used for intelligently analyzing the depth information of the commodity image, extracting image features of the commodity image, and determining attribute information of the commodity according to the image features; an optimal grabbing frame parameter acquisition module is used for analyzing the attribute information of the commodity to generate optimal grabbing frame parameters; a spatial pose acquisition module is used for determining spatial pose data of the mechanical hand in grabbing the commodity according to the optimal grabbing frame parameters; and a grabbing control module is used for generating grabbing control instructions of the mechanical hand according to the spatial pose data, and controlling the mechanical hand to grab the commodity according to the grabbing control instructions. The grabbing success rate and work efficiency in a complex scene are significantly improved.

The present invention discloses a real-time website AI risk assessment and avoidance system and its avoidance method. The system includes a VPN connection module to ensure secure and concealed data capture; a data capture module to collect multiple types of information from the target website in real time; a risk feature extraction module to perform multi-dimensional feature extraction; an AI risk analysis and assessment model to identify and rate risks from multiple aspects such as images and scripts; a risk avoidance and warning module to implement a graded response strategy based on the results; and a feedback and model optimization module to collect feedback and sample optimization models. The method implements website risk avoidance through the steps of VPN connection, data collection, feature extraction, risk assessment, strategy execution, and model optimization. Combining VPN with artificial intelligence technology, the present invention can comprehensively and real-timely assess website risks, dynamically adjust access policies, effectively respond to complex network attacks, and ensure user network access security.

The invention provides a time series data playback method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining playback time, playback starting time and cache time window information, setting a cache time starting point, determining a cache time end point according to the cache time window information, and executing a playback data capturing step. Comprising the following steps: acquiring point location data based on a starting point and an ending point, obtaining a to-be-processed point location data set, sorting the to-be-processed point location data set according to a data acquisition timestamp, updating the timestamp, obtaining a current to-be-played-back data sequence, adding the current to-be-played-back data sequence into a data playback queue, circularly executing a playback data capturing step until the playback ending time is reached, and performing data playback on the current to-be-played-back data sequence. Performing data playback according to the data playback queue; according to the method and the device, the continuity and the integrity of data playback are ensured by dynamically adjusting the data acquisition timestamp, information loss or inaccuracy caused by data segmentation is avoided, then, the change of the data volume can be better adapted by dynamically updating the cache time window, and the resource utilization rate and the stability of the system are improved.

The invention provides an image data processing method and device, equipment, a storage medium and a program product, and the method comprises the steps: responding to a first display interrupt signal arriving according to a preset period and a data capture request, determining a state of a coding operation of first image data triggered based on a second display interrupt signal arriving according to a preset period, the second display interrupt signal arriving earlier than the first display interrupt signal, and the first image data being image data of a first frame stored in a frame buffer queue before the second display interrupt signal arrives; triggering a third display interrupt signal in response to determining that the encoding operation of the state representation for the first image data is completed; and based on the third display interrupt signal, obtaining second image data currently stored in the first frame from the frame buffer queue, and processing the second image data. According to the method, the display interruption can be dynamically adjusted, so that the coding efficiency can be improved, and the frame loss ratio can be reduced.

The invention provides an ISTIO-based content anti-capture method, and relates to the technical field of data security, and the method comprises the steps: building an ANP model, carrying out the dynamic analysis on the basis of the relation of a request state, a duration and an associated request, carrying out the dynamic processing and real-time control through the early warning stage and the blocking recognition stage, and carrying out the real-time analysis of the state of the request, the duration and the associated request. On the basis of supporting blocked dynamic release, fine-grained connection control and release are added, data security is powerfully guaranteed, the security and performance of the system are effectively improved, the workload of personnel intervention is reduced, abnormal requests and data capturing behaviors are dynamically recognized, real-time blocking is carried out, and the data capturing efficiency is improved. And meanwhile, a dynamic release mechanism of blocking behaviors is supported, so that the normal access experience of a user is ensured while the network security is ensured.

The present invention provides a multi-channel data fusion method and system, which relates to data processing technologies. The method is based on a data acquisition unit to capture data from data sources corresponding to each user terminal, obtain acquisition data of multiple dimension features corresponding to each user terminal, construct an initial data column corresponding to each user terminal, perform arrangement and update on the initial data column according to the acquisition data corresponding to each dimension feature to obtain a fusion data column, acquire fusion metrics of multiple dimension features corresponding to the user terminal in the fusion data column, calculate user portrait data corresponding to each user terminal according to the fusion metrics, generate push data corresponding to each user terminal according to the user portrait data, and send the push data to the corresponding user terminal.

The application provides an automobile IBC product offline appearance identification detection system and method, and relates to the technical field of automobile part detection. The system comprises: a feeding vision module for identifying the placement posture of the IBC product to be detected on the conveying line and outputting posture data; a mechanical hand module for grabbing the product according to the posture data and transferring it to a detection station; a rotary table module arranged at the detection station for fixing and rotating the product; a vision detection module for scanning the surface of the product during rotation, acquiring three-dimensional image data, and performing defect detection and classification based on the three-dimensional image data; a data tracing module in communication with a central control system and an enterprise MES system for binding the unique identification of the product and the detection data; and the central control system for coordinating the operation of each module and controlling the mechanical hand module to perform a sorting and stacking operation corresponding to the result according to the result of defect detection and classification. Full automation and full coverage of the detection process are realized.

The present invention belongs to the technical field of data tracking, and provides a system and method for tracking a data crawling path based on object storage, comprising: screening out a data text candidate set based on subject keyword comparison, processing to obtain a retrieval range value based on the data text's domain of origin and the number of texts, judging whether it is necessary to reduce the index range, and if so, reducing the index range by combining a Boolean operator, thereby improving index matching efficiency; utilizing natural language processing technology to obtain semantic vectors of the data text and semantic vectors of the data retrieval text, and processing to obtain similarity; combining feature data of the data text in the corresponding new data text candidate set, and performing correlation analysis to obtain a correlation value, and combining the similarity between the data text and the data retrieval text, processing to obtain a matching degree, and recommending corresponding data text to a user based on the matching degree, thereby improving the crawling accuracy of stored data.

1. Name of the Design Product: Graphical User Interface of Teacher Portrait for Electronic Devices. 2. Use of the Design Product: For electronic devices. 3. Design Key Points of the Design Product: Lie in the graphical user interface in the electronic device. 4. Picture or Photograph that Best Illustrates the Design Key Points: Front View. 5. The electronic device is of conventional design, and other views are omitted. 6. Use of the Graphical User Interface: This design product is used to generate a teacher portrait through data scraping, integration, and analysis, and can intuitively display various information of teachers. 7. Human-Computer Interaction Method of the Graphical User Interface: The front view is used for teacher information display, including basic information, teacher honors, and teacher awards, etc.; Click the "Rank Details" button in the upper left corner of the interface in the front view to jump to the interface change state diagram 1, and the interface change state diagram 1 shows the chart information of the teacher rank; Click or hover the mouse pointer over the year node in the chart in the interface change state diagram 1 to jump to the interface change state diagram 2, and the interface change state diagram 2 shows the corresponding rank information.

The invention discloses a perimeter intrusion monitoring method and system for a tunnel portal high-speed rail line, and relates to the technical field of intelligent monitoring, and the method comprises the steps: obtaining a line tunnel portal sequence driving data set according to a high-speed rail line advancing tunnel portal sequence of a sorted target high-speed rail line, and carrying out the intrusion risk recognition, obtaining a tunnel portal sequence intrusion risk coefficient set; meanwhile, interference analysis is carried out on a high-speed rail line advancing tunnel portal sequence, a tunnel portal sequence interference factor set is obtained, a tunnel portal sequence intrusion risk coefficient set is corrected, and a preset video data capture frequency is determined based on the corrected tunnel portal sequence intrusion risk correction coefficient set; and then differentiated monitoring analysis is carried out on the high-speed rail line tunnel portal, and tunnel portal perimeter intrusion information is determined for early warning management and control. The technical effects that differentiated intelligent monitoring and analysis of the tunnel portal are achieved, the accuracy of intelligent monitoring and analysis and the processing efficiency of intrusion monitoring data are improved, and then the timeliness of early warning of the perimeter intrusion of the tunnel portal is ensured are achieved.

The invention relates to the technical field of digital employee data processing, in particular to a social security digital employee data processing and model building method based on cloud edge collaboration, and the method specifically comprises the following steps: collecting multi-source social security data, and capturing a policy text; constructing a training sample based on the collected multi-source social security data and policy text, and dividing the training sample into a training set, a verification set and a test set in proportion; constructing a social security digital employee question and answer model by adopting a neural network architecture based on an encoder and a decoder, inputting the training set into the model, and processing the training set by the encoder and the decoder to generate an answer sequence; training the model, performing multiple iteration training on the model, ending the training of the model until a preset iteration stopping condition is met, and selecting the model with the best performance through the verification set to be the optimal model; and inputting the test set into the optimal model to generate an answer sequence. According to the method, the social security digital employee question and answer model is constructed, so that the processing efficiency can be improved, the data processing accuracy is enhanced, and the operation cost is reduced.

The present invention provides a health care information push method and system based on social perception technology, the method comprising: obtaining health care comment filter words, and creating an information retriever to crawl comment data according to the health care comment filter words to obtain corresponding health care comment information; performing text processing on the health care comment information to obtain a number of first comment information; performing sentiment analysis on the first comment information to obtain second comment information; obtaining a preset number of the second comment information, combining with a preset reputation value calculation model, to obtain the reputation value of the health care location; generating health care location recommendation information according to the reputation value of the health care location, and pushing the health care location recommendation information to the target user. Therefore, by applying the method described in the present invention, it is possible to effectively promote each health care location to improve its own health care services and assist users in selecting suitable health care locations, and efficiently handle complex health care comments.

The invention discloses a big data network security adaptive defense system based on deep learning, and relates to the technical field of network security, the system comprises a monitoring center, the monitoring center is in communication connection with a deep learning module, a data capture module, a data processing module, a data analysis module and an adaptive defense module; a deep learning model is utilized to quickly identify risks of various network security events possibly occurring in real-time network interaction information, so as to obtain defense modes for each network security event, and meanwhile, the defense intensity of the corresponding defense modes is configured according to risk coefficients of various network security events, so that the defense efficiency is improved. Therefore, the limitation of a single defense mode is avoided, excessive defense is avoided, meanwhile, the adaptive configuration defense system can improve the network security to the maximum extent, and the network security defense efficiency is improved.

The invention discloses a multi-protocol data capturing method and system, and relates to the technical field of data capturing. The multi-protocol data capturing method is applied to the multi-protocol data capturing system, and the multi-protocol data capturing system comprises a client and a server which is in communication connection with the client through multiple network protocols. The multi-protocol data capturing method comprises the following steps: a client sends a data capturing request to a server through one of a plurality of network protocols; the server calls a corresponding processor to perform data capture based on the request type carried in the data capture request; the server converts the grabbed data into a unified intermediate format; and the server converts the data in the unified intermediate format to the client through one of the plurality of network protocols and sends the data to the client. According to the multi-protocol data capturing method and system disclosed by the invention, cross-language and cross-protocol data capturing service can be realized.

An electric vehicle ownership prediction method and system based on multi-source information fusion comprises the following steps: S1, capturing training data from a database by using an automatic data capturing tool and an API interface, and carrying out classification marking on the training data, S2, carrying out data expansion on an original electric vehicle data training set based on a dynamic fuzzy membership driven SMOTE algorithm, and carrying out prediction on the electric vehicle ownership. S3, constructing a six-layer full-connection neural network algorithm, performing data feature extraction on the expanded electric vehicle data training set, and training a feature extraction model, and S4, classifying the electric vehicle data training set after feature extraction based on a random forest classification algorithm of fuzzy set optimization, and training a classifier model. And S5, inputting a new electric vehicle data set sample into the trained model to obtain a classification result, namely an electric vehicle inventory prediction result. The design is not only high in flexibility and adaptability, but also accurate in classification.

The invention relates to the technical field of computers, and discloses an information consultation service platform based on big data analysis and acquisition, which comprises a multi-source data acquisition module, a multi-source data cleaning module, a multi-source data acquisition module, a multi-source data calculation module and a multi-source data integration module, the multi-source data acquisition module acquires multi-source data through a network and an offline survey table, and the multi-source data cleaning module performs formatting cleaning on the classified multi-source data. The multi-source data acquisition module is responsible for acquiring information of a structured data unit, an unstructured data unit, a static data unit, a historical data unit and a user privacy data unit, and the multi-source data calculation module calculates a structured data coverage index # imgabs0 #, a real-time data coverage index # imgabs1 #, a user privacy data coverage index # imgabs2 # and a data fluctuation threshold # imgabs3 #. By calculating the indexes and the threshold values, the problems that data capture of a traditional platform is limited and sudden events cannot be dealt with sufficiently are solved.

The purpose of the present invention is to provide a DFX method of discrete grab bucket based on anchor points, the method comprising: obtaining data grabbing requirements; the logic module configures parameters of the grab template according to the data grabbing requirements; the logic module selects anchor points according to the data grabbing requirements; the anchor points grab data into the grab bucket according to the grab template; and the chip information is analyzed according to the grabbed data. The present invention can grab according to different grabbing conditions in different subsystems and submodules, and can grab multiple times at the same time, solving the problem of timing and directional simultaneous grabbing of internal signals, thereby accelerating the debugging and positioning of chip problems, and greatly improving the efficiency of locating chip problems.

The invention discloses an automatic processing system and method for electric power information communication scheduling operation data. The system comprises a man-machine interaction module, a data capture module, a report processing module, a file management module and a data verification module. The man-machine interaction module is used for realizing data interaction between the system and an operator; the data capturing module is used for automatically extracting work order data through a cross-communication scheduling core business system; the report processing module is used for realizing monthly report updating, daily report generation and end-of-month data archiving; the file management module is used for managing template files, result files and log files related to the system; and the data verification module is used for verifying the accuracy of the data processing result. The method disclosed by the invention is high in data processing efficiency, good in integrity and relatively high in accuracy.