Big data-based information collection system for software development

Abstract

Disclosed in the present invention is a big data-based information collection system for software development. The system comprises: a dynamic data collection module, which is used for automatically collecting a code submission record, a task completion situation and defect tracking information during software development by means of an API interface and a log file; a data pre-processing module, which is used for performing formatting, deduplication and missing value filling processing on collected original data, so as to ensure the quality of the original data for subsequent analysis; a distributed storage module, which is used for using a distributed data storage method to store pre-processed data in a blockchain system, so as to realize efficient storage and access of massive data; and an intelligent analysis module, which is used for using a machine learning algorithm to perform deep analysis on data stored in a distributed manner, so as to identify a software development trend and predict a project development risk. The system can monitor a software development progress in real time, predict a potential problem, and assist in decision making.

Description

An information collection system for software development based on big data Technical Field

[0001] This invention relates to the field of big data technology, and in particular to an information collection system for software development based on big data. Background Technology

[0002] In the field of software development, effective information collection and analysis are crucial to the success of project management. Traditional information collection methods often rely on manual reports and static data analysis, lacking real-time and comprehensiveness, and failing to meet the demands of rapid iterative software development. With the development of big data technology, it has become possible to optimize the software development process using big data techniques.

[0003] Summary of the Invention

[0004] The technical problem to be solved by this invention is how to provide a big data-based information collection system for software development that can monitor software development progress in real time, predict potential problems, and assist in decision-making.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: an information collection system for software development based on big data, the system comprising:

[0006] Dynamic data acquisition module: used to automatically collect code commit records, task completion status and defect tracking information during the software development process through API interface and log files;

[0007] Data preprocessing module: Used to format, deduplicate, and impute missing values ​​in the collected raw data to ensure the quality of the raw data for subsequent analysis;

[0008] Distributed storage module: Employs a distributed data storage method to store pre-processed data in a blockchain system, enabling efficient storage and access to massive amounts of data;

[0009] Intelligent Analysis Module: Used to perform in-depth analysis of distributed storage data using machine learning algorithms, identify software development trends and predict project development risks;

[0010] Visualization and Interaction Module: This module displays the processing results of the intelligent analysis module in the form of data charts and reports for project managers and developers to observe and analyze. It also supports custom queries and alarm settings.

[0011] The beneficial effects of adopting the above technical solution are as follows: The system described in this application provides accurate data support for software development by efficiently and comprehensively collecting and analyzing various types of information in the software development process, thereby improving the efficiency and quality of software development, monitoring the software development progress in real time, predicting potential problems, and assisting in decision-making. Attached Figure Description

[0012] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0013] Figure 1 is a schematic block diagram of the system described in an embodiment of the present invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0015] As shown in Figure 1, this embodiment of the invention discloses an information collection system for software development based on big data, the system comprising:

[0016] Dynamic data acquisition module 101: used to automatically collect code commit records, task completion status and defect tracking information during the software development process through API interface and log files;

[0017] Data preprocessing module 102: used to format, deduplicate, and impute missing values ​​in the collected raw data to ensure the quality of the raw data for subsequent analysis;

[0018] Distributed storage module 103: Employs a distributed data storage method to store pre-processed data in a blockchain system, enabling efficient storage and access to massive amounts of data;

[0019] Intelligent Analysis Module 104: Used to perform in-depth analysis of distributed storage data using machine learning algorithms, identify software development trends and predict project development risks;

[0020] Visualization and Interaction Module 105: This module displays the processing results of the intelligent analysis module in the form of data charts and reports, allowing project managers and developers to observe and analyze them. It also supports custom queries and alarm settings.

[0021] Furthermore, the dynamic data acquisition module automatically collects code commit records, task completion status, and defect tracking information during the software development process using the following methods:

[0022] 1) Information gathering via API interfaces: Query API interface information for software development tools, version control systems (such as Git, SVN, etc.), and defect tracking systems to determine available API interfaces, their parameters, and the format and limitations of the returned data; register an account and obtain API interface access permissions according to the API interface requirements; construct HTTP requests, use programming languages ​​(such as Python) and HTTP client libraries (such as requests) to send requests and obtain responses; parse the data in the API responses to extract the required code commit records, task completion status, and defect tracking information; store the extracted data in a database for subsequent analysis.

[0023] 2) Collect information through log files: Configure logging functionality in software development tools, version control systems, and defect tracking systems to record key operations (such as code commits, task updates, defect reports, etc.) in log files; use scripts or programs to periodically read log files, analyze and extract information related to the software development process; store the data extracted from log files in a unified database for cross-system data analysis and report generation;

[0024] 3) Data synthesis and processing: The information collected through the API interface and the information collected through the log file are synthesized, and the synthesized data is analyzed using data analysis tools to complete the collection of code submission records, task completion status and defect tracking information in the software development process.

[0025] Furthermore, the data preprocessing module's processing method includes the following steps:

[0026] 1) Data Formatting: Convert code commit records, task completion data, and defect tracking information into a standardized format, such as CSV, JSON, or database table format; arrange the data in a certain order according to logical relationships or classification criteria, such as chronological order or category order; add headers and column names to the data tables to explain the meaning and classification of the data; adjust the font color, size, and style in the data tables according to the nature and purpose of the data to enhance readability and visual appeal.

[0027] 2) Data deduplication: Assign a unique identifier (such as UUID) to each piece of data. Before the data is entered into the database, check whether the identifier already exists in the database. If it already exists, delete the data.

[0028] 3) Missing value handling: Calculate the proportion of missing data to the total data. When the proportion of missing data is small and has little impact on the analysis, directly delete the records containing missing values. When a certain data has a large number of missing values ​​and has a significant impact on the analysis objective, use machine learning algorithms to predict missing values ​​and fill them in, so that the data is complete or the number of missing data is reduced to a level that does not affect the data analysis.

[0029] By formatting, deduplicating, and removing missing values ​​from the data collected by the dynamic data acquisition module through the above steps, data quality can be significantly improved, providing a reliable data foundation for subsequent analysis. During the processing, it is necessary to select appropriate methods based on the actual data situation and analytical needs, and to ensure data integrity and consistency.

[0030] Furthermore, the intelligent analysis module processes data using the following method:

[0031] 1) Machine learning algorithm selection and training: Based on the analysis objectives, time series analysis models are used for software development trend identification, and support vector machine models are used for project development risk prediction. In a distributed computing environment, TensorFlow is used for parallel training and optimization of the models to improve training efficiency and model performance.

[0032] 2) Deep analysis and pattern recognition: Analyze data features and extract features useful for software development trends and risk prediction; use machine learning algorithms to analyze code change history to identify defect patterns and user feedback to predict future demand trends;

[0033] 3) Trend prediction and risk assessment: Use trained models to predict the software completion time, future version functional requirements, and technology stack change trends; use machine learning models to assess project development risks, and predict delay risks, increased technical difficulty risks, and insufficient resource risks based on historical data and current project status.

[0034] The intelligent analysis module combines machine learning algorithms to perform in-depth analysis of data stored in distributed storage, which can effectively identify software development trends and predict project development risks, providing strong support for project management and decision-making.

[0035] Furthermore, the visualization and interaction module processes data through the following methods:

[0036] 1) Processing intelligent analysis results: Integrate and clean the data generated by the intelligent analysis module, and use data mining methods to process the integrated and cleaned data to extract valuable information;

[0037] 2) Data visualization: Based on the results of intelligent analysis and display requirements, select the type of chart to be displayed, map the results of intelligent analysis onto the chart and add interactive elements, so that users can dynamically query and analyze data as needed;

[0038] 3) Report generation and custom queries: Design data report templates that integrate visual charts, tables, and text descriptions; utilize data visualization tools to automate report generation; provide custom query functionality to enable users to build query conditions according to their needs and generate personalized data views and reports;

[0039] 4) Alarm settings: Define alarm rules and set alarm notification methods according to business needs and data characteristics. When an alarm occurs, record the detailed information of each alarm and track the alarm events through the alarm tracking module to analyze and process the alarm events.

[0040] The visualization and interaction module presents complex information to users in an intuitive and vivid way, greatly improving user experience and interaction efficiency.

Claims

1. An information collection system for software development based on big data, characterized in that, The system includes: Dynamic data acquisition module: used to automatically collect code commit records, task completion status and defect tracking information during the software development process through API interface and log files; Data preprocessing module: Used to format, deduplicate, and impute missing values ​​in the collected raw data to ensure the quality of the raw data for subsequent analysis; Distributed storage module: Employs a distributed data storage method to store pre-processed data in a blockchain system, enabling efficient storage and access to massive amounts of data; Intelligent Analysis Module: Used to perform in-depth analysis of distributed storage data using machine learning algorithms, identify software development trends and predict project development risks; Visualization and Interaction Module: This module displays the processing results of the intelligent analysis module in the form of data charts and reports for project managers and developers to observe and analyze. It also supports custom queries and alarm settings.

2. The information collection system for software development based on big data as described in claim 1, characterized in that, The dynamic data acquisition module automatically collects code commit records, task completion status, and defect tracking information during the software development process using the following methods: 1) Gather information through API interfaces: Query the API interface information of software development tools, version control systems, and defect tracking systems to determine the available API interfaces, their parameters, and the format and limitations of the returned data; register an account and obtain API interface access permissions according to the API interface requirements; construct HTTP requests, use programming languages ​​and HTTP client libraries to send requests and obtain responses; Parse the data in the API response to extract the required code commit records, task completion status, and defect tracking information; store the extracted data in a database for subsequent analysis. 2) Collect information through log files: Configure logging functionality in software development tools, version control systems, and defect tracking systems to record critical operations in log files; Use scripts or programs to periodically read log files, analyze and extract information relevant to the software development process. Information; data extracted from log files is stored in a unified database for cross-system data analysis and report generation; 3) Data synthesis and processing: The information collected through the API interface and the information collected through the log file are synthesized, and the synthesized data is analyzed using data analysis tools to complete the collection of code submission records, task completion status and defect tracking information in the software development process.

3. The information collection system for software development based on big data as described in claim 1, characterized in that, The data preprocessing module's processing method includes the following steps: 1) Data formatting: Convert code submission records, task completion data, and defect tracking information into a standardized format; arrange the data in a certain order according to the logical relationship or classification criteria; add headers and column names to the data table to explain the meaning and classification of the data; adjust the font color, size, and style in the data table according to the nature and purpose of the data to enhance the readability and visual effect of the data. 2) Data deduplication: Assign a unique identifier to each data entry. Before data is entered into the database, check whether the identifier already exists in the database. If it already exists, delete the data entry. 3) Missing value handling: Calculate the proportion of missing data to the total data. When the proportion of missing data is small and has little impact on the analysis, directly delete the records containing missing values. When a certain data has a large number of missing values ​​and has a significant impact on the analysis objective, use machine learning algorithms to predict missing values ​​and fill them in, so that the data is complete or the number of missing data is reduced to a level that does not affect the data analysis.

4. The information collection system for software development based on big data as described in claim 1, characterized in that, The intelligent analysis module processes data using the following method: 1) Machine learning algorithm selection and training: Based on the analysis objectives, time series analysis models are used for software development trend identification, and support vector machine models are used for project development risk prediction. In a distributed computing environment, TensorFlow is used for parallel training and optimization of the models to improve training efficiency and model performance. 2) Deep Analysis and Pattern Recognition: Analyze data features and extract insights into software development trends and... Useful features for risk prediction; Analyze code change history using machine learning algorithms to identify defect patterns and user feedback to predict future demand trends. 3) Trend prediction and risk assessment: Use trained models to predict the software completion time, future version feature requirements, and technology stack change trends; The project development risks are assessed using machine learning models. Based on historical data and the current project status, the risks of delays, increased technical difficulty, and insufficient resources are predicted.

Images