179 results about "Data scraping" patented technology

Along with promotion of industry 4.0 and rising of intelligent manufacturing, an industrial decision support system plays a more and more important role in improving enterprise operation efficiency and market competitiveness. The invention provides an industrial decision support system based on big data and artificial intelligence technology, and aims to provide accurate decision support for enterprises through deep analysis and real-time processing of massive industrial data. According to the system, by integrating an advanced data acquisition technology, monitoring and data capturing of the whole process of industrial production, including but not limited to production equipment states, process parameters, quality control, supply chain information and the like, are realized. By using artificial intelligence algorithms such as machine learning and deep learning, the system can identify a mode and predict a trend from complex data and construct a corresponding decision model. In addition, a built-in operation optimization engine of the system can provide an optimal solution for production planning, inventory management and resource configuration based on a data analysis result. Through an intuitive user interaction interface, a decision maker can easily obtain key business insights and intelligent decision suggestions so as to make a quick response in a continuously changing market environment. The industrial decision support system not only can improve the production efficiency and the product quality, but also can help an enterprise to reduce the operation cost, avoid potential risks, and promote the enterprise to transform to intelligence and digitization.

The invention discloses a website AI intelligent risk assessment real-time evasion system and an evasion method thereof. The system comprises a VPN connection module to ensure that data capture is safe and hidden; the data capturing module collects various types of information of a target website in real time; the risk feature extraction module performs multi-dimensional feature extraction; the AI risk analysis and evaluation model identifies risks from multiple aspects of images, scripts and the like and grades the risks; the risk avoiding and warning module executes a hierarchical response strategy according to a result; the feedback and model optimization module collects a feedback and sample optimization model. According to the method, website risk avoidance is realized through the steps of VPN connection, data acquisition, feature extraction, risk assessment, strategy execution, model optimization and the like. According to the method, VPN and artificial intelligence technologies are combined, website risks can be comprehensively evaluated in real time, access strategies are dynamically adjusted, complex network attacks are effectively coped, and user network access safety is guaranteed.

The invention discloses an operation and maintenance management system and method for a smart city, and belongs to the technical field of electric digital data processing, and the method specifically comprises the steps: automatically collecting and preprocessing city comprehensive management data through employing an intelligent data capturing technology, building a multi-target garbage station planning model according to the set target and constraint conditions, and carrying out the construction of a multi-target garbage station planning model; solving and generating a preliminary planning scheme through an optimization algorithm; in the scheme evaluation stage, a multi-factor analysis method is adopted to comprehensively view the preliminary scheme, and optimization adjustment is carried out according to the preliminary scheme to obtain a final operation and maintenance scheme; a scheme implementation effect is simulated by means of a three-dimensional modeling and simulation technology, and a feasible implementation scheme is formulated in combination with actual conditions; the intelligent planning of the garbage station is realized, and the urban garbage treatment efficiency and the management level are improved.

The invention belongs to the field of data processing and the field of financial science and technology, and relates to a data processing method based on big data analysis, comprising the following steps: acquiring initial business opportunity data based on a data capture strategy; performing data screening processing on the initial business opportunity data based on an input standard strategy to obtain first business opportunity data; performing data preprocessing on the first business opportunity data to obtain second business opportunity data; acquiring a specified key; screening out a specified encryption algorithm from a plurality of pre-stored encryption algorithms based on the algorithm screening rule; performing encryption processing on the second business opportunity data based on a specified key and a specified encryption algorithm to obtain target business opportunity data; and uploading the target business opportunity data to a preset specified storage medium. The invention further provides a data processing device based on big data analysis, computer equipment and a storage medium. In addition, the invention also relates to a block chain technology, and the target business opportunity data can be stored in a block chain. According to the invention, the leakage risk of the business opportunity data is solved, and the transmission security of the business opportunity data is ensured.

The invention relates to the technical field of crawler algorithms, particularly provides a clothing trend crawler improvement system based on a near-end strategy optimization model, and aims to solve the problem of poor data capture effect of a crawler algorithm in the prior art. The system comprises a keyword receiving unit for receiving clothes keyword information; the reinforcement learning unit is used for adjusting a crawler strategy through a near-end strategy optimization model; a reward mechanism is introduced to reward the clothing data captured by the crawler, and the crawler is driven to capture target clothing data which corresponds to the clothing keyword and has a fashion trend; and the data warehouse construction unit is used for classifying and storing the target clothing data. According to the method, the crawler strategy is optimized and adjusted through the near-end planning optimization model, so that the crawler can dynamically adapt to the change of the Internet environment, data in the fashion clothing industry can be flexibly and efficiently captured and processed, and reliable data support is provided for clothing trend analysis and prediction.

The invention discloses an anti-phishing webpage detection system and method. The anti-phishing webpage detection system comprises a webpage data capture module, a risk analysis module, a malicious webpage judgment module and a response decision module, relates to the technical field of anti-phishing webpage detection. According to the method, multi-dimensional data, including URL information, domain name information and webpage content information, of a webpage are extracted through the webpage data capturing module, a comprehensive original data basis is provided, and accurate support is provided for follow-up risk analysis; according to the method, comprehensive risk analysis is carried out by utilizing various webpage features including URL features, domain name features and content features, and potential phishing webpages can be effectively identified by calculating risk scores of the URL, the domain name and the webpage content; according to the method, the web pages are classified according to the comprehensive risk scores, the high-risk web pages are intercepted, the emergency response logs are generated, the low-risk web pages are continuously monitored, and safe browsing of a user is ensured.

The invention discloses a general news page data acquisition method, device and equipment, a medium and a product, and relates to the technical field of news page data acquisition. The method comprises the following steps: sending an HTTP (Hyper Text Transport Protocol) request to a target news website to obtain return data of a news webpage, calling a browser automation tool Selenium to perform data capture after all elements of the news webpage are loaded when judging that dynamic loading contents exist, and taking a capture result as original data of the news webpage, then, for each node in the DOM tree, calculating to obtain corresponding text density and symbol density, judging whether the webpage content is complex content or not based on a calculation result, and if the webpage content is complex content, analyzing the webpage content through multi-dimensional feature analysis and a big language model used for news page analysis based on rules; and positioning to obtain a final extraction result aiming at the news page key information, and finally carrying out data cleaning and standardization processing on the extraction result to obtain news page data with a uniform format and outputting the news page data.

The data integration method based on the cloud computing technology comprises the steps that a data reading end obtains change data in a source data source through a CDC assembly; the data reading end transmits the first identifier of each batch of changed data to the progress storage end, and transmits the changed data to the data writing end; the progress storage end records a first identifier from the data reading end; the data writing end writes the changed data into the target data source and transmits a second identifier of the data written into the target data source to the progress storage end; the progress storage end records a second identifier from the data writing end; and the data submitting end reads the first identifier and the second identifier recorded by the progress storage end, and submits the second identifier to the change data capturing assembly under the condition that the second identifier is consistent with any one of the first identifiers, so as to indicate the change data capturing assembly not to acquire data related to the second identifier from the source data source. According to the method, the data integration efficiency can be improved.

The invention provides a grabbing control method and device of a robot, the robot, a storage medium and a program product. A first collecting assembly is controlled to collect first sensing data in the current environment where the robot is located, and a mechanical arm is controlled to drive a second collecting assembly to collect second sensing data in the current environment; and controlling the mechanical arm to grab the target object based on the first sensing data and the second sensing data. Thus, the target object can be grabbed according to the first sensing data and the second sensing data obtained by the first collection assembly and the second collection assembly which are different in position, the target object can be grabbed according to data of different view angles, the possibility of misrecognition caused by limited obtained information under a single view angle is reduced, and the recognition accuracy is improved. And the accuracy and the flexibility of the grabbing control method are improved.

The invention discloses an intelligent public opinion research and judgment method and system for public security actual combat, and the method achieves the high-throughput data capture and real-time public opinion inflow through multi-source crawler, breakpoint resume, incremental collection and asynchronous concurrent processing, and greatly reduces the workload of manual screening. A unified, quantifiable and structured public opinion risk grading standard is established, and risk grading standard unification is realized by constructing three types of structured public opinion dictionaries of event subjects, event properties and sensitivity grades; the grading rule is quantified and objective and adapts to the grading system of the public security actual combat, and the subjectivity and inconsistency of research and judgment are remarkably reduced; and the recessive public opinion recognition capability is improved through the deep semantic model. According to the method, the BERT model, keyword vector fusion and the police affair large model are combined, so that accurate analysis of complex semantics such as metaphor, black talk and irony mood is realized; high-robustness semantic matching and duplicate removal are carried out; through accurate extraction of risk factors and event identification, the problem of high missed judgment rate in the prior art is improved.

The invention discloses a manipulator target grabbing path intelligent planning method based on visual depth perception. The manipulator target grabbing path intelligent planning method comprises the following steps that grabbing target point cloud data are obtained and preprocessed to generate a structured point cloud data set; executing SuperPoint division and adjacency relation construction, and generating a SuperPoint graph structure; constructing a plurality of groups of action guide graphs and embedding the action guide graphs into an SPGNet attention mechanism to form an improved SPGNet structure; inputting structured point cloud data, extracting node features and capturing response, and generating a capturing candidate region and a sorting result; determining grabbing point coordinates and grabbing posture parameters according to the sorting result; path search and obstacle prediction are executed in combination with the tail end pose of the mechanical arm, and grabbing path trajectory data are generated; new point clouds are collected in the grabbing process, a structure topology self-adaptive updating unit is called to reconstruct a graph structure, and a final control instruction is output. According to the invention, high-precision grabbing and path self-adaptive control of the manipulator in a complex environment are realized, and the dynamic operation stability is effectively improved.

The invention discloses a depression detection method based on image-text social media analysis, relates to the technical field of automatic detection of depression, and solves the technical problem in the prior art that whether a user is a depression patient or not is detected by analyzing character and picture data published by the user on a social media platform. Specifically, the method comprises four steps of data capture, post screening, image-text representation and fusion classification. Generally speaking, the method comprises the following steps: firstly, capturing a posted document or posted post of a user in a certain time period from a social media platform, including characters and picture data in the posted document; then, performing depression related symptom prediction on the posted document, and screening out the most related K posts; then, respectively carrying out text and picture representation on the screened posts; and finally, fusing the characterization vectors of the two modes for depression detection.

The invention discloses a large model algorithm capability data set construction method and device and electronic equipment, and belongs to the technical field of large models.The method comprises the steps that data capture is conducted from multiple platforms through a data collection module, and original data are obtained; performing data cleaning on the original data by using a data cleaning module to obtain target cleaned data; performing correctness verification on the target cleaning data by using a correctness verification module to obtain verification data; performing algorithm labeling on the verification data by using an algorithm labeling module to obtain labeled data; performing data standardization on the annotated data by using a data standardization module to obtain standardized data; and evaluating the standardized data by using the evaluation framework module to obtain an evaluation result. According to the method, wide data coverage and high quality are ensured; advantages and disadvantages of the model in different algorithm types can be deeply analyzed; the evaluation result is more accurate; an interactive tool is provided, and the use cost of a user is greatly reduced.

The invention provides a manipulator commodity grabbing system and method based on visual recognition, and relates to the technical field of equipment control, and the method comprises the steps: collecting the image information of a commodity in a working area based on a multi-modal visual collection device, analyzing the commodity image information, and obtaining the depth information of a commodity image; the intelligent detection module is used for intelligently analyzing the depth information of the commodity image, extracting image features of the commodity image and determining attribute information of the commodity according to the image features; the optimal grabbing box parameter acquisition module is used for analyzing the attribute information of the commodity to generate optimal grabbing box parameters; the spatial pose acquisition module is used for determining spatial pose data of the manipulator during commodity grabbing according to the optimal grabbing frame parameters; and the grabbing control module is used for generating a grabbing control instruction of the manipulator according to the spatial pose data, and controlling the manipulator to grab the commodity according to the grabbing control instruction. And the grabbing success rate and the operation efficiency in a complex scene are remarkably improved.

The invention discloses a target information tracing method for internet big data, which comprises the following steps: capturing data from a plurality of internet data sources by using a distributed data capturing tool, collecting the data to a data processing platform, and preprocessing the captured data to generate a feature vector about the data; carrying out normalization processing and quantum state mapping on the data, and encoding the data into quantum state representation through a parameterized quantum circuit; constructing a quantum data index structure based on a quantum variation algorithm, and generating a quantum data index tree; adding a federal learning mechanism in the quantum data index tree, performing cooperative training on quantum state representations from different data sources, and updating global model parameters to obtain optimized quantum state representations; and inputting the optimized quantum state representation into the quantum neural network, and calculating and outputting a traceability result through forward propagation. According to the invention, efficient and accurate traceability of target information in an internet big data environment is realized, and the reliability of information traceability is improved.

A method of extracting and conditioning content for use in a hierarchically structured merchant risk monitoring environment may include receiving an indication of a merchant under consideration, identifying one or more web pages likely to have information associated with travel exposure for the merchant under consideration based on the indication, conducting data scraping on the identified one or more web pages to attempt to obtain scraped travel exposure data, responsive to obtaining the scraped travel exposure data, employing a machine learning module to determine a travel risk exposure estimate based on the scraped travel exposure data, and responsive to not obtaining the scraped travel exposure data, determining the risk exposure estimate based on a predefined travel risk estimate associated with an industry classification of the merchant under consideration.

The invention belongs to the technical field of data acquisition, and discloses a method for realizing data capture in combination with a browser plug-in, the scheme is based on Chrome plug-in extension and realizes target website data capture through a mode of simply configuring an address of a website needing to be captured and a data interface URL (Uniform Resource Locator), and the scheme comprises the following specific steps: step 1, task plan configuration; step 2, task information configuration; step 3, executing a capturing task; 4, capturing and storing data; and 5, uploading the data. According to the method, the real Chrome browser is started and the Tampermonkey plug-in is combined for crawling, an anti-crawling strategy can be effectively bypassed, the data capturing efficiency is improved, compared with a traditional method, according to the scheme, all data of the server are configured through the configuration file, the target website data can be rapidly collected in a specified mode, operation is easy, and the data capturing efficiency is improved. And the learning cost is reduced.

The invention belongs to the technical field of data safety, and particularly relates to a visual safety brain modeling and early warning platform of an expressway toll collection system, which comprises a data capturing module used for acquiring real-time interaction data and a toll station three-dimensional model of the expressway toll collection system, and constructing an expressway toll station panoramic three-dimensional model in a three-dimensional environment; the dynamic display module comprises a first floating information board, a second floating information board, a third floating information board, a fourth floating information board and a fifth floating information board; the early warning module is used for analyzing the real-time interaction data acquired by the data capturing module to find abnormal data and tracing an abnormal event; and the adjusting module provides a dynamic adjusting mechanism according to the matching result. According to the method, the distribution strategy is adjusted in real time to cope with changes of emergency situations and tasks, the empty window period occurring in the adjustment process is avoided as much as possible, and network security management vulnerabilities of the highway toll collection system can be prevented.

The application provides a mechanical hand commodity grabbing system and method based on visual recognition, and relates to the technical field of equipment control. The system comprises the following steps: collecting image information of commodities in a working area based on a multi-modal visual collection device, analyzing the commodity image information, and obtaining depth information of the commodity image; an intelligent detection module is used for intelligently analyzing the depth information of the commodity image, extracting image features of the commodity image, and determining attribute information of the commodity according to the image features; an optimal grabbing frame parameter acquisition module is used for analyzing the attribute information of the commodity to generate optimal grabbing frame parameters; a spatial pose acquisition module is used for determining spatial pose data of the mechanical hand in grabbing the commodity according to the optimal grabbing frame parameters; and a grabbing control module is used for generating grabbing control instructions of the mechanical hand according to the spatial pose data, and controlling the mechanical hand to grab the commodity according to the grabbing control instructions. The grabbing success rate and work efficiency in a complex scene are significantly improved.

The present invention discloses a real-time website AI risk assessment and avoidance system and its avoidance method. The system includes a VPN connection module to ensure secure and concealed data capture; a data capture module to collect multiple types of information from the target website in real time; a risk feature extraction module to perform multi-dimensional feature extraction; an AI risk analysis and assessment model to identify and rate risks from multiple aspects such as images and scripts; a risk avoidance and warning module to implement a graded response strategy based on the results; and a feedback and model optimization module to collect feedback and sample optimization models. The method implements website risk avoidance through the steps of VPN connection, data collection, feature extraction, risk assessment, strategy execution, and model optimization. Combining VPN with artificial intelligence technology, the present invention can comprehensively and real-timely assess website risks, dynamically adjust access policies, effectively respond to complex network attacks, and ensure user network access security.

The invention provides a time series data playback method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining playback time, playback starting time and cache time window information, setting a cache time starting point, determining a cache time end point according to the cache time window information, and executing a playback data capturing step. Comprising the following steps: acquiring point location data based on a starting point and an ending point, obtaining a to-be-processed point location data set, sorting the to-be-processed point location data set according to a data acquisition timestamp, updating the timestamp, obtaining a current to-be-played-back data sequence, adding the current to-be-played-back data sequence into a data playback queue, circularly executing a playback data capturing step until the playback ending time is reached, and performing data playback on the current to-be-played-back data sequence. Performing data playback according to the data playback queue; according to the method and the device, the continuity and the integrity of data playback are ensured by dynamically adjusting the data acquisition timestamp, information loss or inaccuracy caused by data segmentation is avoided, then, the change of the data volume can be better adapted by dynamically updating the cache time window, and the resource utilization rate and the stability of the system are improved.

The invention provides an image data processing method and device, equipment, a storage medium and a program product, and the method comprises the steps: responding to a first display interrupt signal arriving according to a preset period and a data capture request, determining a state of a coding operation of first image data triggered based on a second display interrupt signal arriving according to a preset period, the second display interrupt signal arriving earlier than the first display interrupt signal, and the first image data being image data of a first frame stored in a frame buffer queue before the second display interrupt signal arrives; triggering a third display interrupt signal in response to determining that the encoding operation of the state representation for the first image data is completed; and based on the third display interrupt signal, obtaining second image data currently stored in the first frame from the frame buffer queue, and processing the second image data. According to the method, the display interruption can be dynamically adjusted, so that the coding efficiency can be improved, and the frame loss ratio can be reduced.

The invention provides an ISTIO-based content anti-capture method, and relates to the technical field of data security, and the method comprises the steps: building an ANP model, carrying out the dynamic analysis on the basis of the relation of a request state, a duration and an associated request, carrying out the dynamic processing and real-time control through the early warning stage and the blocking recognition stage, and carrying out the real-time analysis of the state of the request, the duration and the associated request. On the basis of supporting blocked dynamic release, fine-grained connection control and release are added, data security is powerfully guaranteed, the security and performance of the system are effectively improved, the workload of personnel intervention is reduced, abnormal requests and data capturing behaviors are dynamically recognized, real-time blocking is carried out, and the data capturing efficiency is improved. And meanwhile, a dynamic release mechanism of blocking behaviors is supported, so that the normal access experience of a user is ensured while the network security is ensured.

The present invention provides a multi-channel data fusion method and system, which relates to data processing technologies. The method is based on a data acquisition unit to capture data from data sources corresponding to each user terminal, obtain acquisition data of multiple dimension features corresponding to each user terminal, construct an initial data column corresponding to each user terminal, perform arrangement and update on the initial data column according to the acquisition data corresponding to each dimension feature to obtain a fusion data column, acquire fusion metrics of multiple dimension features corresponding to the user terminal in the fusion data column, calculate user portrait data corresponding to each user terminal according to the fusion metrics, generate push data corresponding to each user terminal according to the user portrait data, and send the push data to the corresponding user terminal.

The application provides an automobile IBC product offline appearance identification detection system and method, and relates to the technical field of automobile part detection. The system comprises: a feeding vision module for identifying the placement posture of the IBC product to be detected on the conveying line and outputting posture data; a mechanical hand module for grabbing the product according to the posture data and transferring it to a detection station; a rotary table module arranged at the detection station for fixing and rotating the product; a vision detection module for scanning the surface of the product during rotation, acquiring three-dimensional image data, and performing defect detection and classification based on the three-dimensional image data; a data tracing module in communication with a central control system and an enterprise MES system for binding the unique identification of the product and the detection data; and the central control system for coordinating the operation of each module and controlling the mechanical hand module to perform a sorting and stacking operation corresponding to the result according to the result of defect detection and classification. Full automation and full coverage of the detection process are realized.

The present invention belongs to the technical field of data tracking, and provides a system and method for tracking a data crawling path based on object storage, comprising: screening out a data text candidate set based on subject keyword comparison, processing to obtain a retrieval range value based on the data text's domain of origin and the number of texts, judging whether it is necessary to reduce the index range, and if so, reducing the index range by combining a Boolean operator, thereby improving index matching efficiency; utilizing natural language processing technology to obtain semantic vectors of the data text and semantic vectors of the data retrieval text, and processing to obtain similarity; combining feature data of the data text in the corresponding new data text candidate set, and performing correlation analysis to obtain a correlation value, and combining the similarity between the data text and the data retrieval text, processing to obtain a matching degree, and recommending corresponding data text to a user based on the matching degree, thereby improving the crawling accuracy of stored data.

1. Name of the Design Product: Graphical User Interface of Teacher Portrait for Electronic Devices. 2. Use of the Design Product: For electronic devices. 3. Design Key Points of the Design Product: Lie in the graphical user interface in the electronic device. 4. Picture or Photograph that Best Illustrates the Design Key Points: Front View. 5. The electronic device is of conventional design, and other views are omitted. 6. Use of the Graphical User Interface: This design product is used to generate a teacher portrait through data scraping, integration, and analysis, and can intuitively display various information of teachers. 7. Human-Computer Interaction Method of the Graphical User Interface: The front view is used for teacher information display, including basic information, teacher honors, and teacher awards, etc.; Click the "Rank Details" button in the upper left corner of the interface in the front view to jump to the interface change state diagram 1, and the interface change state diagram 1 shows the chart information of the teacher rank; Click or hover the mouse pointer over the year node in the chart in the interface change state diagram 1 to jump to the interface change state diagram 2, and the interface change state diagram 2 shows the corresponding rank information.

The invention discloses a perimeter intrusion monitoring method and system for a tunnel portal high-speed rail line, and relates to the technical field of intelligent monitoring, and the method comprises the steps: obtaining a line tunnel portal sequence driving data set according to a high-speed rail line advancing tunnel portal sequence of a sorted target high-speed rail line, and carrying out the intrusion risk recognition, obtaining a tunnel portal sequence intrusion risk coefficient set; meanwhile, interference analysis is carried out on a high-speed rail line advancing tunnel portal sequence, a tunnel portal sequence interference factor set is obtained, a tunnel portal sequence intrusion risk coefficient set is corrected, and a preset video data capture frequency is determined based on the corrected tunnel portal sequence intrusion risk correction coefficient set; and then differentiated monitoring analysis is carried out on the high-speed rail line tunnel portal, and tunnel portal perimeter intrusion information is determined for early warning management and control. The technical effects that differentiated intelligent monitoring and analysis of the tunnel portal are achieved, the accuracy of intelligent monitoring and analysis and the processing efficiency of intrusion monitoring data are improved, and then the timeliness of early warning of the perimeter intrusion of the tunnel portal is ensured are achieved.

The present invention relates to the technical field of talent recruitment platform, specifically a self-service interactive platform for skilled talent recruitment, which is composed of a data capture module, a three-level semantic network model module, a person-job matching module, a skilled talent selection module, a skilled talent big data map generation module, a skilled talent directory generation module, and a skilled talent portrait generation module. By using the three-level semantic network model module, the captured raw data can be analyzed and learned at a deeper level to generate knowledge graphs and person-job portraits. This in-depth data processing provides the platform with a comprehensive understanding of talents and positions, thereby improving the accuracy of matching. The explicit features, potential features, ripple features, and intersection features, these four dimensions conduct a comprehensive analysis of talents, not only limited to the experience and skills on the resume, but also to tap the potential capabilities of talents and the ripple effects that may be caused to the team.

The invention relates to the technical field of digital employee data processing, in particular to a social security digital employee data processing and model building method based on cloud edge collaboration, and the method specifically comprises the following steps: collecting multi-source social security data, and capturing a policy text; constructing a training sample based on the collected multi-source social security data and policy text, and dividing the training sample into a training set, a verification set and a test set in proportion; constructing a social security digital employee question and answer model by adopting a neural network architecture based on an encoder and a decoder, inputting the training set into the model, and processing the training set by the encoder and the decoder to generate an answer sequence; training the model, performing multiple iteration training on the model, ending the training of the model until a preset iteration stopping condition is met, and selecting the model with the best performance through the verification set to be the optimal model; and inputting the test set into the optimal model to generate an answer sequence. According to the method, the social security digital employee question and answer model is constructed, so that the processing efficiency can be improved, the data processing accuracy is enhanced, and the operation cost is reduced.