67 results about "Big data architecture" patented technology

A system and method for rapid development and deployment of reusable analytic code for use in computerized data modeling and analysis is provided. The system includes a centralized, continually updated environment to capture pre-processing steps used in analyzing big data, such that the complex transformations and calculations become continually fresh and accessible to those investigating business opportunities. The system incorporates deep domain expertise as well as ongoing expertise in data science, big data architecture, and data management processes. In particular, the system allows for rapid development and deployment of analytic code that can easily be re-used in various data analytics applications, and on multiple computer systems.

A method and system of on spot diagnosis of one or more issues associated with a machine includes collecting sound data and image data associated with a machine through a mobile device and transmitting them to a cloud server over a communications network. The on spot diagnosis includes analyzing the sound data and image data in combination with an on spot feedback system through the cloud server. The on spot feedback system is communicatively coupled to a machine-learning engine and a Big Data architecture. The machine issue condition is indicated through a user interface dynamic such a circular gauge. An alarm is set, through a mobile device, for the machine issue.

The invention discloses a power distribution network multi-dimensional visualization platform based on a big data architecture, and relates to the power distribution network technology field; data processing of a data integration and management module, a data storage module, a data fusion module, a power distribution network data processing module, a power distribution network comprehensive earlywarning management module and a visual large-screen comprehensive display module is realized through a communication technology; power distribution regulation and control are improved and upgraded, and real-time data acquisition and monitoring in a power distribution network are realized; the invention discloses a power distribution network multi-dimensional visualization platform constructed based on a geographic information system; global observability of the distribution network is realized; based on a big data analysis architecture, the network distribution, a real-time operation state, ageographic position and other information of a power distribution network can be directly displayed via visual display in the form of a dynamic or static chart, and a multifunctional application platform capable of realizing operation maintenance, analysis management, intelligent prediction and the like is formed; and the problems that the existing power distribution network supervision is mainlyused for a decentralized working mode and the operation process is more difficult and complex are solved.

The invention provides an analysis index calculation platform based on KKS coding rule and big data structure. A receiver module screens out usable information from massive data, pretreats the data through a stream preprocessing module and groups the data through a data distribution module, thereby facilitating the development of big data processing work; a data computing module integrates the complex algorithm into an integrated algorithm model, and a data publishing module distributes the indexed data to the corresponding database for storage, which can provide clear and intuitive results; aprocess management and control module is responsible for managing the circulation and transmission of data information in the entire platform, connecting all modules in series, and the system management module is used to send the indexed data to the application display platform for display, so that different work systems can be combined organically, thus realizing comprehensive management. Through the above design, the power plant monitoring system is more perfect, and can meet more use needs.

The invention provides a multi-source data processing method, device and system for a big data architecture and a block chain. The method comprises the following steps: performing data acquisition ona plurality of data sources and converting the acquired data into a data stream with a unified format; realizing classified caching of the data streams and providing a data stream output interface; acquiring a data stream through a data stream output interface and calling a big data open source algorithm to consume the acquired data stream; and obtaining a data stream through the data stream output interface and storing the obtained data stream to the block chain. The invention provides a unified and lightweight data processing platform capable of meeting various actual business scenes, data acquisition of different data sources is realized, and the acquired data is converted into a data stream with a unified format, so that various data query and analysis tools can be conveniently and quickly read. In addition, the classified and stored data streams can be quickly and conveniently transferred to the block chain, so that the application of the block chain is met.

The invention provides a real-time data platform based on KKS coding rule and big data structure, which uniformly codes all monitoring indexes through a coding management system, and provides a set ofuniversal codes for the whole platform, and each power plant uses the same codes, thereby avoiding the problem that data cannot be smoothly docked. Data monitoring and processing system can monitor many equipments in many power plants at the same time, and cache the monitoring data through the way of establishing database, so it can deal with massive data efficiently and in high real-time. Data output management system achieves the information output and release to different targets through the shared channel, which effectively improves the compatibility of the platform. Through the above design, the problems existing in the use of the traditional data monitoring system are effectively solved, so that the data monitoring system is more perfect and can meet more use needs.

The invention belongs to the technical field of data computing processing, and discloses a performance service system and a performance service method based on a big data architecture. The performanceservice method comprises the steps of: storing applied data in a big data platform, synchronizing transaction data in real time by the big data platform; re-calculating a staff performance commissionwage after a user modifies a commission scheme at any time. The performance service method is characterized in that: re-calculation of data during the time period such as the current day, the currentmonth and the current season is supported; micro service architecture development is adopted, and a development mode of domain model driving design is used to perform high cohesion and low coupling extension. Through adoption of the performance service method of the invention, a performance statistic result can be calculated in real time, delay of a system can not be caused by data size, and performance commission data during the time period exceeding one season or half a year can be quickly calculated after the user modifies the commission scheme or closes the accounting.

The invention discloses a security and high-efficiency vulnerability management system, and relates to the technical field of vulnerability management,, in particular to a security and high-efficiencyvulnerability management system, a threat intelligence system and an internet intelligence intelligent collection system based on big data architecture, which are mainly used for uniformly integrating intelligence searched by company self-research, friend-merchant sharing, industry sharing and the internet, and sharing related intelligence to a third-party system or product according to industrycharacteristics. Te local threat and vulnerability management platform is mainly used for realizing Internet asset auditing, scanning engine centralized management and vulnerability management supportfunctions. According to the safe and efficient vulnerability management system, except for collecting and analyzing the risk data of the scanning missing product, other risk data can be imported forunified analysis, and the management capacity of quick response, ordered repair and continuous optimization is provided for the customer vulnerability management process in combination with local riskcontinuous monitoring and vulnerability management full-process management and control capacity.

The invention discloses a big data framework-based ecological environment monitoring grid system. The system comprises an environment monitoring device and a service running module; and a multi-environment parameter access module is arranged at one end of the environment monitoring device. The big data framework-based ecological environment monitoring grid system is provided with an internet-of-things data collection module, a heterogeneous data processor, a lustre-based environment big data analysis system, ecological environment monitoring quality evaluation method and model, and an ecological environment grid system; protocol contents are configurable; no secondary development is required; a multisource heterogeneous data processing method provides a one-stop data access service; multi-monitoring object-based ecological environment quality comprehensive evaluation is realized through the ecological environment monitoring quality evaluation method and model; wisdom management and application of ecological environment monitoring is realized through the grid application system; use convenience is brought for people; and better use prospects are achieved.

A big data architecture software testing method and device, a computer apparatus and a storage medium are disclosed, wherein the big data architecture software is based on a Hadoop framework, and themethod comprises the steps of obtaining a test data table require for performing a functional test task; analyzing attributes of the test data table to obtain corresponding data construction parameters; inputting the data construction parameter into a preset data conversion script to generate a corresponding data construction command; converting the data in the test data table according to the data construction command to obtain test data; performing functional tests on corresponding tasks of the big data architecture software by using the test data. The big data Hadoop framework testing method is first proposed in the present application, and prior to this, there is no relevant t method; the method can be widely applied to the test of big data projects in various groups, so as to facilitate the test of the big data Hadoop framework.

The invention discloses a method for accessing a medical institution system to an intelligent medical cloud service platform, and belongs to the technical field of cloud computing. The method comprises the following steps: functions and network slices are divided; a terminal of a medical institution system user completes authentication through an authentication center, and data interaction with anMEC host is achieved, so that data of the medical institution system is stored in a functional data storage node in the MEC host through a data recombination node and a data opening node; data in each functional data storage node is based on corresponding management between functions and network slices, and the network slices of corresponding levels are transmitted to the intelligent medical cloud service platform. The method is designed by utilizing a network slice (NS), a mobile edge computing (MEC) technology and a big data Hadoop architecture, and is a new method for accessing the medicalinstitutions to the intelligent medical cloud platform, namely a method for realizing butt joint between each medical institution system and the intelligent medical cloud service platform.

The invention relates to a data storage system and a data storage method of large data. The system comprises a four-layer structure. The metadata structure layer defines and stores a plurality of metadata types and attributes of the metadata types. The metadata layer defines and stores all the association relations between metadata object information and metadata object according to the attributesof metadata type. The data structure layer defines and stores the structural relationship of statistical data according to the metadata object information. The data layer stores statistical data according to the structural relationship of the statistical data. The invention is based on DDI metadata standard, Defines a four-tier structure for statistical data storage: metadata structure, Metadata,data structure and data, so as to unify the storage and management of massive micro-data according to the big data structure, break down the barriers of data resource integration, realize the flexible and efficient storage and use of statistical data, and then largely solve the problems of large amount of data, complex structure, rapid growth rate, and data analysis and use difficulties and so on.

The invention belongs to the technical field of Internet of Vehicles information communication, and discloses an Internet of Vehicles information security transmission and intelligent early warning system, which comprises vehicle navigation equipment for processing data; a memory card which is connected with the vehicle-mounted navigation equipment through a USB interface and comprises a USB driveprogram, a WiFi drive program and a service entry program; a mobile phone terminal and a service terminal are provided with a 3G / 4G module and a WiFi module, and the terminals are in wireless communication connection with the one-key terminal; and a server background which is in communication connection with the mobile phone through a 3G or 4G network. According to the method, a method system forlarge-scale heterogeneous data processing is applied, security management in the big data era is inevitably data-driven, and security management capabilities such as information acquisition, data fusion, event storage, advanced security analysis, situation awareness and visualization must be reconstructed on the basis of a big data technology by taking a big data architecture as a support.

The invention provides an application display platform based on KKS coding rule and big data structure, which obtains real-time data or historical data to be displayed through a data calling module, thereby solving the problem of single data source. The control part management module selects the control part for visualization processing according to the need, and combines the selected control parts by the interface management unit to configure the data display interface, so that the user can define the data display interface by himself, so that the data display interface can be configured freely, adjusted flexibly and adapted to different use situations; the data visualization module converts the data into a chart form which can reflect the data information intuitively, and it can displaythe data to the user through a variety of display terminals, which is more convenient and flexible. Through the above design, the problems existing in the use of the application display platform are effectively solved, and the application display platform is more perfect and can meet more use requirements.

The invention provides a water conservancy and hydropower big data architecture building method. In the method, a water conservancy and hydropower big data collection system, a water conservancy and hydropower big data mining system, a water conservancy and hydropower big data mining model base, a water conservancy and hydropower big data analysis system and a water conservancy and hydropower big data distributed server node controlling system are contained, and the method comprises the steps of available data collection and storage, water conservancy and hydropower big data mining, correlation data extraction and water conservancy and hydropower big data analysis. Compared with a traditional big data analysis method, by means of the water conservancy and hydropower big data architecture building method, water conservancy and hydropower industry data can be subjected to systematized collection, mining and analysis so as to form guiding data, the construction performance and the quality control level of water conservancy and hydropower engineering are easily improved, and meanwhile, the information management level of the water conservancy and hydropower engineering is improved.

The invention discloses an Internet of Vehicles data analysis platform based on a big data architecture. The platform comprises a data receiving module, a data analysis task management module and a data pushing module, the data receiving module receives the collected data reported by a vehicle-mounted terminal, judges and processes the collected data according to a certain data preprocessing logic, and writes the data meeting a certain condition into message middleware Kafka; the data analysis task management module submits a data analysis task to a Spark distributed computing engine, performsunified monitoring and management in the execution process of the data analysis task, and writes monitoring indexes and a final analysis result in a database Clickhouse in the operation period of thedata analysis task; the data pushing module receives a request that a user subscribes a certain data analysis task to output a result based on a Websocket protocol; and the problems that a traditional data analysis platform cannot be connected with an online big data scene environment, and the model training analysis process is not standard are solved.

The invention belongs to the technical field of big data storage, and particularly relates to a big data multi-level storage architecture which comprises three levels, namely a first level, a second level and a third level, wherein the hierarchy I comprises the following steps: only storing hot spot data; an Hbase is used for providing low-delay random reading and writing; an SSD hard disk is usedas hardware to provide good performance; the hierarchy II comprises the following steps: storing relative popular data; spark is used for providing efficient large-scale analysis and calculation; themethod comprises the following steps of: storing data by using an open-source Apache Hadoop distributed file system; and a Parquet format is used for data storage, so that the Spark calculation efficiency is improved. According to the method, a multi-level big data architecture platform is built by utilizing different big data frameworks and technologies, and different technologies and solutionsare applied to different use scenes, so that the service performance is improved, and the calculation time and the hardware cost are reduced.

The invention discloses a precise network attack detection and early warning platform. The precise network attack detection and early warning platform comprises a multi-engine sandbox subsystem, an auxiliary detection subsystem and a data analysis subsystem; the data analysis subsystem comprises a correlation analysis system and a big data architecture-based data analysis system, and the correlation analysis system comprises a sandbox intermediate result correlation module, an alarm rule correlation module, an alarm and flow correlation module and a flow modeling analysis module; the auxiliarydetection subsystem comprises an AV module and an IDS module, According to the platform, a layered defense structure is adopted to divide the system into an auxiliary detection subsystem system whichfocuses on known threat detection, a multi-engine sandbox detection subsystem which focuses on malicious behavior detection and a data analysis subsystem; for each stage of the APT attack, the accurate network attack detection and early warning platform can perform effective detection and timely early warning on the APT attack in the stages of single-point breakthrough, back door installation, secret channel establishment, data stealing and the like.

A semiconductor fabrication scheduling method includes creating a load scheduling data schema including facility data of product lots to be dispatched to a plurality of workstations; generating a load schedule profile using a load-balancing model and based on the load scheduling data schema, wherein the load-balancing model includes one or more objective functions and there is at least one weight factor in an objective function; generating a current load schedule based on the load schedule profile; dispatching the product lots to the plurality of workstations using the current load schedule to complete fabrication of the product lots; obtaining a set of current key performance indicators (KPIs) of the completed fabrication of the product lots; and automatically adjusting the weight factors of the objective functions of the load-balancing model based on the current KPIs using a big-data architecture to generate a next load schedule for next cycle of fabrication.

The invention discloses a data intelligent analysis visualization system and method. The system comprises a data transmission module, a database visualization module, a report design module, an intelligent interaction analysis module, a data structure feedback module and a data visualization display screen module. The system is reasonable in design, sorting can be carried out based on query results, sorting is carried out based on dimensions of the system, sorting is carried out on dimensions based on sizes of summary indexes, sorting is carried out based on formula values, and data results are sent to a data visualization display screen through a multi-dimensional analysis unit and a drilling analysis unit. On the basis of an overall big data architecture, step-by-step data drilling and computational analysis are carried out through a drilling analysis unit to obtain corresponding underlying data, results of various single tables are selected and placed in the same area through a result splicing mode, formula operation and display are carried out according to the results, and multiple complex tables are obtained; report links can be established for combination and association to obtain various associated reports.

The invention discloses an internet advertisement risk monitoring method and system based on a big data architecture. The method comprises the following steps: periodically acquiring data from the Internet; accessing the acquired data in real time, and performing verification cleaning and picture identification on the data; storing and warehousing the accessed data; analyzing and extracting suspected illegal data in the stored data in combination with a preset illegal behavior feature library; and based on the analyzed and extracted data, carrying out query service and business application ofsuspected illegal data. According to the method, an intelligent supervision mode is fully utilized, supervision means are enriched, the supervision mode is optimized, the supervision effect is improved, a long-acting mechanism is perfected, benign development of Internet advertisements is accelerated and promoted, and the supervision level is greatly improved.

The invention discloses a log data modeling method and system, and the system comprises a client and a log generalization unit, and the client is in communication connection with the log generalization unit. And the log generalization unit is set to be used for acquiring the log uploaded by the client and performing internal processing on the log uploaded by the client. According to the method and the system provided by the invention, modeling processing is performed on a large amount of log data through log generalization, so that later data retrieval and analysis are facilitated. And meanwhile, the log generalization process only depends on strategy configuration, and the strategy configuration can be managed on an interface, so that personalized customization of various types of log generalization processes can be conveniently supported. In the log generalization process, secondary processing on the generalization result is supported, and the scene that secondary processing needs to be carried out on the original log content can also be met. According to the method, big data architecture deployment is adopted, high performance and high throughput are guaranteed, and the requirements of large data volume and high performance requirements in actual application scenes are met.

The invention belongs to the technical field of environment monitoring, and discloses an ecological environment monitoring gridding system based on big data architecture and a monitoring method thereof. The system comprises a ZigBee communication module, a 5G signal communication module, an environment information detection module, an information collection module, a state indication module, a cloud server module, an environment state evaluation module, an early warning module and a man-machine interaction module. According to the invention, synchronous monitoring can be carried out on environmental information of multiple different positions at the same time, unified coordination management is facilitated, the environmental problem disposal efficiency is improved, data storage is carriedout through the cloud server, the data storage capacity and storage stability are improved, historical data can be conveniently checked, and data change analysis can be conveniently carried out; the data can be effectively simplified by performing preliminary screening processing on the collected data, the data transmission burden is reduced, and the transmission efficiency and the subsequent processing efficiency are improved, so that the response speed of the system is improved.

The invention provides a big data teaching management architecture system, and relates to the technical field of big data architecture. The big data teaching management architecture system comprises aserver cluster, a switch and a virtual desktop controller, wherein the server cluster is connected with a disk array through a storage switch; the server cluster is connected with a virtual machine management platform; the server cluster is connected with each of a manager server and the virtual desktop controller through the switch; and the virtual desktop controller is connected with each of anintelligent terminal, a notebook, a personal computer and a thin client through an LAN. The big data teaching management architecture system is capable of providing safe and reliable experiment environment and greatly enhancing big data skill; and by adoption of desktop virtualization platform architecture, the big data teaching management architecture system can have a complete whole-series virtualization scheme, excellent user experience, more comprehensive data safety guarantee, a centralized WEB management mode and a special local service mode.

The invention discloses a power simulation method based on a large data architecture. The method comprises the following steps that basic data of a power system state is combined; data is preprocessedto form a space vector model used for representing an object state; large data is analyzed and processed; the visual processing of analysis results is carried out and the results are shown. By integrating a large data technology and a power grid simulation technology organically, a power simulation calculation platform based on a large data structure is formed, more and more data are added, the calculation speed is increased, practical problems in the operation process of the power grid can be found more quickly, the problems can be identified correctly and solved in time, so that the operation of the power grid is more stable and safe, and the operating cost of the power grid is reduced.

The invention discloses a fence real-time early warning method and system based on a big data technology. The method comprises the following steps: customizing an electronic map fence and a corresponding early warning rule; collecting vehicle driving data uploaded by a vehicle terminal in real time based on a big data architecture to obtain a vehicle driving data set; taking a self-defined electronic map fence, a corresponding early warning rule and the vehicle driving data uploaded by the vehicle terminal as an analysis data set, extracting feature data, matching a corresponding analysis algorithm model, performing matching analysis based on the vehicle driving data set and the early warning rule through an analysis score model, and judging whether to trigger the early warning rule set by a user; and pushing real-time warning information to the vehicle triggering the early warning rule. According to the invention, an electronic map fence mode is adopted, vehicle information is collected in real time through big data, and barrier-free early warning of vehicles driving to the area is realized through a self-defined early warning rule.

The invention discloses a big data architecture-based data processing method, apparatus and device, and a storage medium, which are applied to a big data architecture. The architecture is configured with multiple processing functions; and the multiple processing functions have the same core code modules. The method comprises the steps of calling the multiple processing functions to perform presetprocessing on obtained incremental data, thereby obtaining processing results; and storing the processing results in a preset storage layer. According to the method, the apparatus, the device and thestorage medium, the multiple processing functions for processing the incremental data in the architecture have the same core code modules, namely, the multiple processing functions in the architecturehave unified logic layers, so that the multiple processing functions are ensured to be same in logic, the system development and maintenance costs can be reduced, the system maintenance difficulty islowered, and the quality difference of the processing results of processing layers is avoided.

The invention discloses a cloud platform based on a custom equipment protocol library and a big data architecture. The cloud platform mainly comprises a cloud platform management program, a storm-based streaming big data processing program, a data presentation program, a docker hub mirror image warehouse, a distributed message queue kafka cluster and a distributed cache redis. The cloud platform has the advantages of being high in universality, automation degree and expandability, simple and convenient to deploy and high in data real-time performance. The cloud platform provides an own equipment library and an own acquisition data transmission protocol, and utilizes a storm-based streaming big data processing program to process and display monitoring data in real time. For an acquisition instrument and a sensor based on RS485 communication, a template language for describing an acquisition command and a return result of the acquisition instrument and the sensor is self-defined by the cloud platform. Besides, the cloud platform also has the capability of automatically deploying and running an acquisition program on the industrial personal computer by configuring information of equipment such as a sensor, an acquisition instrument and the industrial personal computer in a management program of the cloud platform.

The invention provides a pushing method of medical-field we-media platform data under a big-data architecture. The method includes the following steps: step 1, data selection and transformation, wherein valid data are extracted from user behavior log information, and transformed into modeling data matrices; step 2, evaluation data matrix construction, wherein evaluation data matrices meeting recommendation model robustness evaluation are screened out from the modeling data matrices; step 3, recommendation model construction and estimation; and step 4, recommendation result combination and formation. According to the method, systematic analysis and modeling are carried out on the medical-field we-media platform relevant-data, an estimation method of model prediction is given, valid recommendation can be carried out for doctor reading on the basis of the medical-field we-media platform data through a model, a doctor is enabled to acquire relevant literature in real time when logging in a platform, and a basis is provided for achieving the best reading selection.