170 results about "Open source data" patented technology

Systems and methods (e.g., utilities) for use in providing automated, lightweight collection of online, open source data which may be content-based to reduce website source bias. In one aspect, a utility is disclosed for use in extracting content of interest from at least one website or other online data source (e.g., where the extracted content can be used in a subsequent search query). In other aspects, utilities are disclosed that are operable to perform various types of analyses on such extracted content and present graphical representations of such analyses on a display of a client device.

Systems and methods (e.g., utilities) for use in providing automated, lightweight collection of online, open source data which may be content-based to reduce website source bias. In one aspect, a utility is disclosed for use in extracting content of interest from at least one website or other online data source (e.g., where the extracted content can be used in a subsequent search query). In other aspects, utilities are disclosed that are operable to perform various types of analyses on such extracted content and present graphical representations of such analyses on a display of a client device.

The present invention discloses a multidimensional online analytical processing (MOLAP)-based data processing method and apparatus. The data processing method comprises the steps of creating a data cube according to a fact table and a dimension table; performing data pre-calculation on all possible combinations of dimensions according to data recorded in the data cube; and storing a pre-calculation result in an open-source database, so as to determine an inquire result according to the pre-calculation result in inquiry. Through adoption of the method, the existing data inquiry solution can be optimized, and non-technical staff can inquire in massive data.

The invention discloses a multi-mode heterogeneous power big data fusion analysis management system which comprises a multi-source data acquisition layer for data collection of multiple data sources,a data interface layer which is corresponding to multiple data types and is provided with multiple data interfaces, a power big data fusion management subsystem which includes multiple open source databases, a big data analysis engine which includes an open source algorithm library based on a Spark parallel computing framework and achieves calculation and analysis of stored data in the open sourcedatabases by calling a corresponding analysis method in the open source algorithm library, and a model access interface layer which is connected to the big data analysis engine and the open source database and includes multiple access interfaces, wherein the access to the open source algorithm library and the open source database is achieved by the corresponding access interfaces. According to the system, the performance of data processing can be improved, a data value is fully exploited and the data asset management is realized, and a fast and efficient data management guarantee is providedfor an upper power professional model.

The invention provides a pedestrian re-identification method and device based on a semi-supervised training mode and a storage medium. The method comprises the following steps: constructing a trainingsample set by using the collected target domain sample set and open source data set based on a semi-supervised training mode; and training a deep residual network by using the training sample set toobtain a trained deep residual network model, identifying the acquired pedestrian images by using the trained deep residual network model to obtain feature values of the pedestrian images, and determining whether the pedestrian images belong to the same person according to cosine distances among the feature values. According to the invention, the virtual sample is generated; a smoothing function is constructed when a virtual sample is generated; and meanwhile, a DBSCAN clustering algorithm is used for adding pseudo tags to the virtual samples, local features and global features are used in thedeep neural network, and a joint loss function of different weight combinations is adopted, so that the trained deep neural network is accurate and more reliable in recognition.

The invention discloses an intelligent tour route planning auxiliary method based on internet word frequency. The method comprises basic data preparation and organizing, tourism development condition determination, and overlay analysis-guided optimal tour route planning. The method uses division boundaries of a specific city or region as a work base map; collects names of tour sights and potential sights and organizes a property list; takes the Baidu Travel, Baidu Search, Baidu open source development platform and other open source data on the internet as main data sources; is mainly based on internet word frequency; furthermore carries out place name parsing of the sight names by means of the Baidu LBS open platform; by virtue of GIS software, locates accommodation facilities, such as standby sights and hotels, and corrects coordinates of the facilities; overlays obtained word frequency statistics data and other information; determines the property type and development direction of each sight, gives suggestions to tour route organization and tour service center address selection, etc.; and finally creates an optimal tour route.

The invention relates to a hybrid collaborative filtering movie recommendation model based on a bilateral network structure, and belongs to the field of recommendation systems. The model introduces anidea of deep learning and uses posters and description texts of movies, effective feature expression of users and articles is generated through CNN, a probability matrix decomposition technology is used to introduce potential users and movie models through Gaussian noise parameters, and a deep neural recommendation model with a bilateral network structure is integrated and built. The model is applied to open source dataset of ML_1M expressed by MovieLens_1M from a MovieLens website to perform experiments, so that it can be verified that when the deep learning is used to introduce auxiliary information of movie posters and description texts for building the hybrid collaborative filtering recommendation model, under the condition that the sparse interactive scoring matrix of the users and articles is improved, whether or not the influence of the data sparsity on recommendation accuracy is reduced. In the case of sparse data, the hybrid collaborative filtering movie recommendation modelbased on the bilateral network structure has the advantages that the ability to generate effective feature expression is more obvious, the influence of data sparsity is reduced, and the problem of inaccurate recommendation is effectively improved.

The invention relates to a software recommendation method capable of effectively combining software projects, which are currently developed by a developer, based on group comments. The software recommendation method comprises the following steps: obtaining open source data, and pre-processing the projects and comment document tags; extracting local project characteristics of a user, and matching the projects with comment documents by utilizing the comment document tags and titles and the project names and tags; calculating the similarity and the correlation between a local project and an acquired open source project; and performing linear weighted summation of the similarity and the correlation, and generating a project recommendation list for the user. The software recommendation method disclosed by the invention can be integrated with a software development environment of the developer; the retrieval efficiency of the developer on related software in an internet range can be increased; and technical support can be obtained or software reuse can be realized from the recommended software projects.

Described is a disease prediction system using open source data. The system includes a preprocessing module, a learning module, and a prediction module. The preprocessing module receives a dataset of N trend results related to a disease event and generates an enhanced filter signal (EFS) curve related to the disease event. The learning module receives the EFS curve and generates a predicted number of cases of the disease event and, using a plurality of machine learning methods, generates a plurality of predictions that the disease event will happen within a future time period. The prediction module determines precision and recall for each of the plurality of predictions and, based on the precision and recall, provides a likelihood that the disease event will occur.

The invention discloses a secure data storage method based on a distributed system infrastructure. The method comprises a first step of generating digest information; a second step of encrypting data; a third step of hiding a random secret key; a fourth step of storing a ciphertext; and a fifth step of storing related information. In the first step, data to be stored is signed by an information digest algorithm so as to generate the digest information; in the second step, the data to be stored is encrypted by the random secret key generated by a secret key generation function so as to acquire corresponding ciphertext; in the third step, the random secret key generated in the second step is hidden so as to acquire random secret key information of the ciphertext; in the fourth step, the ciphertext acquired in the second step is stored in a distributed file system; and in the fifth step, the digest information generated in the first step, the random secret key information acquired in the third step and file names are stored in an open source database. The invention also discloses a secure data read method based on the distributed system infrastructure.

The invention provides an automatic data query, synchronization and storage method. A database, an open source data storage tool and a data caching tool are provided. The automatic data synchronization method comprises the following steps of obtaining database change data obtained by an analysis information unit, judging whether automatic synchronization service information is subscribed, if yes, generating change information, making a message queue unit distribute business information to an information synchronization processing unit to carry out data synchronization on the business information distributed by the message queue unit, and if no, finishing generation of the change information. According to the automatic data synchronization system, the updating data in the database can be automatically captured and be synchronized to the open source data storage tool, so that the development cost of developers is saved, synchronization and check of the open source data storage tool are not needed, each link of the system is configurable, and new business and old business can be flexibly added and deleted according to the configuration.

The invention relates to a threat information IOC credibility analysis method, which comprises the steps of obtaining threat information IOC, analyzing and establishing network threat behavior activities and communities, and dynamically optimizing an established model at preset time intervals. In the present invention, attackers and data source features are obtained by deeply mining flow data andlog data of own equipment, third-party shared data and data in each project through technical means. By establishing a threat intelligence IOC credibility scoring model, the data use of each data source can be met, data source credibility verification in an existing threat information source sharing system is satisfied, the problems that a large amount of threat intelligence source data is inaccurate and threat intelligence lacks a failure mechanism in the existing market are solved, creditworthiness evaluation can be carried out no matter whether a three-party open source data source is collected or flow analysis is carried out on own equipment, the system can dynamically adjust the model structure along with time, and the data failure mechanism is ensured.

The invention provides a public opinion information dynamic monitoring method. The method comprises the steps that open source data collection is conducted according to a source website provided by a user and thematic information, and the obtained webpage data is stored to the local; the webpage data obtained in the collection process is analyzed, diversified Internet information is subjected to normalization processing, prophase network page data is stored in a file form, and the analyzed result is subjected to database storage. According to the public opinion information dynamic monitoring method, the data collection process based on a search engine is perfected, and the public opinion information is monitored in real time by means of the efficient data mining algorithm.

The invention discloses an enterprise risk evaluation system and method based on open source data. The system comprises: a data crawling module which can crawl data related to a to-be-evaluated enterprise from a webpage; a data word segmentation module which can perform word segmentation processing on the crawled data and count word frequency; a scoring module which is used for judging according to segmented words obtained by word segmentation processing and word frequencies thereof, wherein the scoring module comprises a plurality of sub-modules, each sub-module evaluates one aspect of factors influencing the development prospect of the enterprise to finally obtain a reasonable enterprise score, the risk of the enterprise can be judged according to the score, and judgment and comparison can also be carried out according to the difference of the scores of different enterprises.

The invention provides an automatic data inquiry synchronous storage system, which comprises a database, an open source data storage tool, a data cache tool, a message queue unit, an analysis information obtaining unit and an information synchronous processing unit, wherein the open source data storage tool is used for fast storing the complicated and diversified data to be inquired; the data cache tool is used for caching the data in the inquiry process; the message queue unit is used for performing queue processing on change information; the analysis information obtaining unit is used for obtaining database change, generating the change information, delivering the analysis data and outputting the data to the message queue unit; the message synchronous processing unit is used for performing the synchronous processing on the message data of the message queue unit and then inputting the data into the open source data storage tool. The automatic data synchronous system can automatically capture the updating data in the database and synchronizing the data to the open source data storage tool; the development cost of developers is reduced; the open source data storage tool synchronizing and checking are not needed; the configurable effect of each link of the system is achieved; the new and old services can be flexibly added and deleted according to configuration.

The invention discloses a medical question-answering system construction method based on a language model and entity matching. The method comprises the steps: S1, data collection; S2, deep neural network model design; S3, named entity recognition model training and knowledge graph construction; S4, complete medical retrieval type question-answering system construction. The method specifically comprises: collecting network medical discussion posts, cleaning the network medical discussion posts, and storing the cleaned network medical discussion posts into ElasticSearch to serve as a retrieval data set; processing open source data of the competition data set by using a medical natural language, and training a named entity recognition model related to medical treatment; and collecting a public data set of the open source website to form a medical knowledge graph so as to expand a retrieval process. According to the medical question-answering system method based on language model and entity matching, after the question-answering system is constructed and recalled, finely arranged and comprehensively scored, the most appropriate answer is output in combination with a reasonable scoring mechanism, and the defects of a retrieval type question-answering system and a knowledge graph type question-answering system are overcome.

The invention discloses a method for recognizing settlement parking spaces meeting the social parking requirement on the basis of open source data. The method includes the following steps of firstly, mining and arranging research data; secondly, conducting address analysis on employment data through the Baidu LBS open platform; thirdly, conducting primary judgment, wherein spatial positioning is conducted on data of all types in a traffic zone base map through a GIS, and row coordinate correction and collection are conducted on the data; fourthly, conducting secondary judgment, wherein potential districts meeting the job-house matching requirement are obtained by screening spatial units with the job-house ratios higher than a preset value and spatial units with the employee number distribution densities higher than the whole-region average density; fifthly, recognizing settlement parking space distribution meeting the social parking requirement and the coverage of the settlement parking space distribution. The potential matching districts of cities are recognized through the open source data, and therefore settlement parking spaces can meet the social parking requirement, the social cost spent when an information issuing party searches potential target settlements is saved, and the space resource optimization in the staggered parking process is assisted.