Data processing method and related apparatus

Abstract

A data processing method and related apparatus are provided in embodiments of the present disclosure. The method includes: obtaining, based on a first number of instances of a first cloud-based microservice, a plurality of unified abstract syntax trees (ASTs) corresponding to a plurality of fragments of a target code; and loading, based on a second number of instances of a second cloud-based microservice, the plurality of unified ASTs into a graph database as a code property graph (CPG) of the target code; where the first number is configurable based on a workload requirement corresponding to the first cloud-based microservice, and / or, the second number is configurable based on a workload requirement corresponding to the second cloud-based microservice. The separation of the first cloud-based microservice and the second cloud-based microservice makes it possible to separately configure their respective instances, that is, the first number and the second number are both configurable, separate configuration for the instances of the two cloud-based microservices may be beneficial for improving scalability, resource utilization and latency.

Description

DATA PROCESSING METHOD AND RELATED APPARATUSTECHNICAL FIELD[00011 The present disclosure relates to the field of software engineering and computer science technologies, and in particular, to a data processing method and related apparatus.BACKGROUND|0002] When using a Static Code Analyzer to analyze source codes, it may be required to edit and execute DSL queries as fast as possible, in order to provide analysis and recommendations to developers on the fly. Currently, in order to execute a DSL query, the Static Code Analyzer needs to either build a CPG from scratch, or load a previously built database for the whole project from disk.

[0003] This background information is provided to reveal information believed by the applicant to be of possible relevance to the present disclosure. No admission is necessarily intended, nor should be construed, that any of the preceding information constitutes prior art against the present disclosure.SUMMARY

[0004] In a first aspect, a data processing method is provided by the present disclosure, and the method includes:obtaining, based on a first number of instances of a first cloud-based microservice, a plurality of unified abstract syntax trees (ASTs) corresponding to a plurality of fragments of a target code; andloading, based on a second number of instances of a second cloud-based microservice, the plurality of unified ASTs into a graph database as a code property graph (CPG) of the target code;where the first number is configurable based on a workload requirement corresponding to the first cloudbased microservice, and / or, the second number is configurable based on a workload requirement corresponding to the second cloud-based microservice.

[0005] According to the embodiments of the present disclosure, the plurality of unified ASTs corresponding to a plurality of fragments of a target code are obtained based on a first number of instances of a first cloud-based imicroservice, and then the plurality of unified ASTs are loaded into the graph database as a CPG of the target code. The separation of the first cloud-based microservice and the second cloud-based microservice makes it possible to separately configure their respective instances, that is, the first number and the second number are both configurable, separate configuration for the instances of the two cloud-based microservices may be beneficial for improving scalability, resource utilization and latency. Since the first number and the second number are configurable, so the computing resources (nodes) are properly shared among microservices, this can ensure minimal (near-real-time) latency of the updates, both to the database and the results of the code analysis. The proposed microservice architecture can work on multiple machines (or pods speaking in Kubemetes language), thus providing a possibility to use different machines for the same analysis in parallel, thereby solving the scalability problem. Besides, different services can be scaled according to workload, thereby achieving load balancing. In addition, based on the solution proposed in the present disclosure, a real time database can be obtained and analysis results updates can be achieved, having in memory service it is possible to fit into a target latency using flexible service with real time scalability.

[0006] In a possible implementation of the first aspect, updating, based on a third number of instances of a third cloud-based microservice, the CPG of the target code;where the updating of the CPG of the target code includes at least one of following operations: adding at least one extra edge or graph for the CPG of the target code;updating at least one property of at least one node in the CPG of the target code; orperforming de-duplication on the CPG of the target code.

[0007] In some cases, the CPG graph, as built from the unified ASTs, may be missing certain information (such as missing nodes and / or edges) or, alternatively, have certain information in the de-normalized state. Further, invocations of the same C++ or Java function may produce different “function call” nodes which would contain references to the different (but semantically identical) “function declaration” nodes. Therefore, in these cases, it is possible to call the third cloud-based microservice to update the CPG. The updating of the CPG of the target code may include one or more of: adding at least one extra edge or graph for the CPG of the target code, the at least one extra edge may include, e.g., call edge, refersTo edges, the at least one graph may include one or more data flow graphs (DFG), one or more code property graphs (CPG); updating at least one property of at least one node in the CPG of the target code, for example, the at least one property of the at least one node may be a de-normalized state of each of the at least one node; or performing de-duplication on the CPG of the target code, for example, two invocations of the same C++ or Java function may produce two “function call” nodes which would contain referencesto the two different (but semantically identical) “function declaration” nodes, in this case, it is possible to call the third cloud-based microservice to update the CPG to remove the duplication. Various ways of updating are provided, so that the flexibility for updating the CPG is improved and the accuracy of the CPG is increased.

[0008] In a possible implementation of the first aspect, the third number is configurable based on a workload requirement corresponding to the third cloud-based microservice, and the workload requirement corresponding to the third cloud-based microservice includes a maximum number of instances allocable for a client from which the target code is submitted and used for running the third cloud-based microservice.

[0009] In a possible implementation of the first aspect, the workload requirement corresponding to the first cloud-based microservice includes a maximum number of instances allocable for a client from which the target code is submitted and used for running the first cloud-based microservice.

[0010] In a possible implementation of the first aspect, the workload requirement corresponding to the second cloud-based microservice includes a maximum number of instances allocable for a client from which the target code is submitted and used for running the second cloud-based microservice.

[0011] In the proposed solution, the numbers of parallel instances of different microservices and the database space usage per client are configurable, which makes it possible for multiple users to work with the same project simultaneously, without creating the same database for each user. Besides, it allows to save money to user providing flexible resource management on demand. Further, it allows to create better user experience by fixing latency allowing to reach real time analysis as you type in IDE for example and make more predictable service behavior.

[0012] In a possible implementation of the first aspect, the configuration of the first number and / or the second number is performed locally at a client from which the target code is submitted. In this way, the client (end user) may locally configure the first number and / or the second number, and submit the target code to the cloud, so that the cloud could deal with the target code based on the first number and / or the second number configured by the client, thereby allowing for customization and optimization of the development process according to the client’s specific needs and environment, as well as saving computing resource of the client.

[0013] In a possible implementation of the first aspect, at least one of following settings is configurable for the graph database:a space usage for the CPG in the graph database;a number of instances used for running nodes of the CPG; ora number of replicas of the graph database allocable for a client from which the target code is submitted.

[0014] The resource consumed by the CPG can be configured in various ways, allowing for controlled development of the CPG, this flexibility makes the graph database more adaptive, especially when it is used to store CPGs for multiple clients.

[0015] In a possible implementation of the first aspect, configuration of the first number and / or the second number is based on a size of the target code. In a possible implementation of the present disclosure, the configuration of the third number is also based on a size of the target code. When the target code occupies a large space, it would be more reasonable to allocate more computing resources for handling this target code, so more instances for building the CPG of the target code (i.e., the first and second numbers) and more instances for updating the CPG may be configured, thereby improving the adaptivity of the whole service for handing different sizes of target codes. The splitting of the whole service into separate microservices (the microservice architecture) makes it possible to scale these microservices independently of each other, accommodating for the size of a particular analysis task.|0016] In a possible implementation of the first aspect, before the obtaining of the plurality of unified ASTs, the method further includes:downloading the target code based on a link, where the link indicates a location where the target code is stored or a location where a patch of the target code is stored;splitting the target code to obtain the plurality of fragments.

[0017] In a possible implementation of the first aspect, the method further includes:receiving at least one domain specific language (DSL) query request from at least one corresponding client;for each of the at least one DSL query request, analyzing the graph database based on a CPG in the graph database corresponding to the DSL query request to obtain a corresponding DSL query result, where data in the DSL query result is in a static analysis results interchange format (SARIF).

[0018] In the proposed solution, CPG graph database is running as a microservice, the DSL query request (request service) is independent from CPG creation and modifications, having CPG graph database as a separate service makes independent DSL queries and CPG loading. In this way, a user can load CPG once and then execute DSL queries at any moment, which creates better user experience in development environment. Further, the DSL query request(s) from one or more clients may be handled separately, and the DSL query result may be returned in a format of SARIF, which facilitates further processing by providing a standardized representation of static analysis results, enabling easy integration with various development tools and improving interoperability across differentsystems.

[0019] In a possible implementation of the first aspect, the at least one DSL query request comprises multiple DSL query requests, and at least one of the multiple DSL query requests is submitted locally by a corresponding client based on an integrated development environment (IDE) instance at the corresponding client. When there are multiple DSL query requests, one or more of the multiple DSL query requests can be submitted via one or more IDE instances. Submitting the DSL query request locally from an IDE instance would enhance performance, reduce latency by processing requests on the client side, leading to faster response times.

[0020] In a possible implementation of the first aspect, the analyzing of the graph database is based on at least one of following settings:a size of computing resources allocated for the analyzing;a size of memories allocated for the analyzing; or,a number of DSL query requests executable within a predefined time period.

[0021] By configuring the computing resources, such as CPU and memory, for carrying out DSL analysis, the overall flexibility and efficiency of the system can be significantly improved.

[0022] In a possible implementation of the first aspect, the first cloud-based microservice is an AST parser service, and the second cloud-based microservice is an AST loader service.

[0023] In a second aspect, a data processing apparatus is provided by the present disclosure, and the apparatus includes:an obtaining module, configured to obtain, based on a first number of instances of a first cloud-based microservice, a plurality of unified abstract syntax trees (ASTs) corresponding to a plurality of fragments of a target code; anda loading module, configured to load, based on a second number of instances of a second cloud-based microservice, the plurality of unified ASTs into a graph database as a code property graph (CPG) of the target code;where the first number is configurable based on a workload requirement corresponding to the first cloudbased microservice, and / or, the second number is configurable based on a workload requirement corresponding to the second cloud-based microservice.

[0024] In a possible implementation of the second aspect, the apparatus further includes:an updating module, configured to update, based on a third number of instances of a third cloud-based microservice, the CPG of the target code;PC17RU2024 / 000341where the updating of the CPG of the target code includes at least one of following operations: adding at least one extra edge or graph for the CPG of the target code;updating at least one property of at least one node in the CPG of the target code; orperforming de-duplication on the CPG of the target code.

[0025] In a possible implementation of the second aspect, the third number is configurable based on a workload requirement corresponding to the third cloud-based microservice, and the workload requirement corresponding to the third cloud-based microservice includes a maximum number of instances allocable for a client from which the target code is submitted and used for running the third cloud-based microservice .

[0026] In a possible implementation of the second aspect, the workload requirement corresponding to the first cloud-based microservice includes a maximum number of instances allocable for a client from which the target code is submitted and used for running the first cloud-based microservice.

[0027] In a possible implementation of the second aspect, the workload requirement corresponding to the second cloud-based microservice includes a maximum number of instances allocable for a client from which the target code is submitted and used for running the second cloud-based microservice.

[0028] In a possible implementation of the second aspect, configuration of the first number and / or the second number is performed locally at a client from which the target code is submitted.

[0029] In a possible implementation of the second aspect, at least one of following settings is configurable for the graph database:a space usage for the CPG in the graph database;a number of instances used for running nodes of the CPG; ora number of replicas of the graph database allocable for a client from which the target code is submitted.

[0030] In a possible implementation of the second aspect, configuration of the first number and / or the second number is based on a size of the target code.

[0031] In a possible implementation of the second aspect, the obtaining module is further configured to: download the target code based on a link, where the link indicates a location where the target code is stored or a location where a patch of the target code is stored;where the apparatus further includes a splitting module, configured to:split the target code to obtain the plurality of fragments.

[0032] In a possible implementation of the second aspect, the apparatus further includes a receiving module,configured to:receive at least one domain specific language (DSL) query request from at least one corresponding client; where the apparatus further includes an analyzing module, configured to:for each of the at least one DSL query request, analyze the graph database based on a CPG in the graph database corresponding to the DSL query request to obtain a corresponding DSL query result, where data in the DSL query result is in a static analysis results interchange format (SARIF).

[0033] In a possible implementation of the second aspect, the at least one DSL query request comprises multiple DSL query requests, and at least one of the multiple DSL query requests is submitted locally by a corresponding client based on an integrated development environment (IDE) instance at the corresponding client.

[0034] In a possible implementation of the second aspect, the analyzing of the graph database is based on at least one of following settings:a size of computing resources allocated for the analyzing;a size of memories allocated for the analyzing; or,a number of DSL query requests executable within a predefined time period.

[0035] In a possible implementation of the second aspect, the first cloud-based microservice is an AST parser service, and the second cloud-based microservice is an AST loader service.

[0036] In a third aspect, a data processing apparatus is provided by the present disclosure, and the apparatus is configured to execute the method according to the first aspect or any possible implementation of the first aspect.

[0037] In a fourth aspect, a computing device cluster including at least one computing device is provided by the present disclosure, and the computing device includes at least one processor and a memory coupled with the at least one at least one processor, where the memory stores instructions which, when executed by the at least one processor, cause the computing device cluster to execute the method according to the first aspect or any possible implementation of the first aspect.

[0038] In a fifth aspect, a computer-readable medium is provided by the present disclosure, and the computer-readable medium stores computer execution instructions which, when run on a computing device cluster, cause the computing device cluster to execute the method according to the first aspect or any possible implementation of the first aspect.

[0039] In a sixth aspect, a computer program product is provided by the present disclosure, and the computer program product includes computer execution instructions which, when run on a computing device cluster, causethe computing device cluster to execute the method according to the first aspect or any possible implementation of the first aspect.

[0040] In a seventh aspect, a computer program is provided by the present disclosure, and the computer program includes computer execution instructions which, when run on a computing device cluster, cause the computing device cluster to execute the method according to the first aspect or any possible implementation of the first aspect.

[0041] Embodiments of the present disclosure provide a data processing method and related products. Aplurality of unified ASTs corresponding to a plurality of fragments of a target code are obtained based on a first number of instances of a first cloud-based microservice, and then the plurality of unified ASTs are loaded into the graph database as a CPG of the target code. The separation of the first cloud-based microservice and the second cloud-based microservice makes it possible to separately configure their respective instances, that is, the first number and the second number are both configurable, separate configuration for the instances of the two cloud-based microservices may be beneficial for improving scalability, resource utilization and latency. Since the first number and the second number are configurable, so the computing resources (nodes) are properly shared among microservices, this can ensure minimal (near-real-time) latency of the updates, both to the database and the results of the code analysis. The proposed microservice architecture can work on multiple machines (or pods speaking in Kubemetes language), thus providing a possibility to use different machines for the same analysis in parallel, thereby solving the scalability problem. Besides, different services can be scaled according to workload, thereby achieving load balancing. In addition, based on the solution proposed in the present disclosure, a real time database can be obtained and analysis results updates can be achieved, having in memory service it is possible to fit into a target latency using flexible service with real time scalability.BRIEF DESCRIPTION OF DRAWINGS

[0042] The accompanying drawings are used to provide a further understanding of the present disclosure, constitute a part of the specification, and are used to explain the present disclosure together with the following specific embodiments, but should not be construed as limiting the present disclosure.

[0043] FIG. 1A is a schematic illustration of an exemplary scenario for accessing a cloud-based service according to one or more embodiments of the present disclosure.

[0044] FIG. IB is a schematic illustration of another exemplary scenario for accessing a cloud-based service

[0045] FIG. 2 is a schematic illustration of an exemplary microservice architecture according to one or more embodiments of the present disclosure.(0046] FIG. 3 is a flowchart of a data processing method according to one or more embodiments of the present disclosure.

[0047] FIG. 4 is a schematic illustration of settings for cloud-based microservices according to one or more embodiments of the present disclosure.

[0048] FIG. 5 is a schematic illustration of an exemplary pipeline for provision of a cloud-based microservices according to one or more embodiments of the present disclosure.

[0049] FIG. 6 is a block diagram of a data processing apparatus according to one or more embodiments of the present disclosure.

[0050] FIG. 7 is a schematic structural diagram of a computing device according to one or more embodiments of the present disclosure.

[0051] FIG. 8 is a schematic diagram of an architecture of a computing device cluster according to one or more embodiments of the present disclosure.

[0052] FIG. 9 is a schematic diagram of a connection between computing devices over a network according to one or more embodiments of the present disclosure.DESCRIPTION OF EMBODIMENTS10053] In the following description, reference is made to the accompanying figures, which form part of the present disclosure, and which show, by way of illustration, specific aspects of embodiments of the present disclosure or specific aspects in which embodiments of the present disclosure may be used. It is understood that embodiments of the present disclosure may be used in other aspects and include structural or logical changes not depicted in the figures. The following detailed description, therefore, is not to be taken in a limiting sense, and the scope of the present disclosure is defined by the appended claims.

[0054] Before elaborating the embodiments of the present disclosure, several terms will be clarified in the first place.

[0055] Static Code Analyzer: a computer program which performs analysis of source code statically, i.e. without actually executing it.merging at least one of its following items:abstract syntax trees (AST),control-flow graphs (CFG), andprogram dependence graph (PDG).

[0057] A CPG can be queried using a Domain Specific Language (CPG DSL for short):functionDeclaration fd wherefd contain fimctionCall fc whereand (fc.name match "bar" ,fc.arguments.size() = 2,fc.type.name = "short",fc.function.name = "bar"fc.function.parameters[0].name — "a",fc.function.parameters[l].name = "b"fc.function.parameters[0].type.name = "int" fc.function.parameters[l].type.name = "long",fc.fiinction.type.name = "short");

[0058] Unified Abstract Syntax Tree (AST): a unified AST is a language-independent AST model which can equally represent the code written in Python, C++, or Java, using a union of syntactic constructs available in all of these languages.

[0059] Sharding: a type of data partitioning which separates large data sets into smaller, faster and more easily manageable parts. Sharding can be implemented by some processing unit. It should be noted that there is no limitation on the way how sharding is done, the implementation of sharding could depend on the data to be split.

[0060] Apache Kafka: a distributed data store optimized for ingesting and processing streaming data in real¬ time. More details would be found at the following site: “https: / / kafka.apache.org / ”.

[0061] A Graph Database: a specialized database that uses graph structures for semantic queries with nodes, edges, and properties to represent and store data. Graph Databases can natively store CPG graphs. And support Gremlin or Cyther as a query language.

[0062] SARIF : an industry standard to exchange and aggregate the output of Static Code Analyzers. SARIF is JSON-based.

[0063] JSON: a lightweight data-interchange format.

[0064] Cloud: computing services offered by third-party providers over the network.

[0065] Kubemetes, also known as K8s, is an open source system for automating deployment, scaling, and management of containerized applications.

[0066] Containerized applications: applications run in isolated packages of code called containers. Containers include all the dependencies that an application might need to run on any host operating system, such as libraries, binaries, configuration files, and frameworks, into a single lightweight executable.

[0067] Pods: the smallest deployable units of computing that you can create and manage in Kubemetes.

[0068] Version control is a system that records changes to a file or set of files over time so that you can recall specific versions later.

[0069] Version control system is a software tool that automates version control.

[0070] Source code branch: a set of files under version control may be branched or forked at a point in time so that, from that time forward, two copies of those files may develop at different speeds or in different ways independently of each other, i.e., you diverge from the main line of development and continue to do work without messing with that main line.

[0071] Git: a distributed version control system that tracks versions of files. It is often used to control source code by programmers collaboratively developing software.

[0072] When using a Static Code Analyzer to analyze source codes, it may be required to edit and execute DSL queries as fast as possible, in order to provide analysis and recommendations to developers on the fly. Currently, in order to execute a DSL query, the Static Code Analyzer needs to either build a CPG from scratch, or load a previously built database for the whole project from disk, which is time consuming. Besides, when a developer makes a small change to the source code, the Static Code Analyzer needs to rebuild the CPG for the whole project, which is time consuming.

[0073] Further, for each project analyzed, it’s necessary to start a new JVM process, which will allocate a different amount of memory, so it is hard to effectively manage resource allocation. Besides, using native libraries loaded into the JVM process of the Static Code Analyzer introduces risks of failures and crashes in case of multithreaded access to the native code.

[0074] For example there are some existing solutions like CodeFuse, CodeQL, VSCode, and etc.

[0075] For CodeFuse, as can be seen from CodeFuse instruction, this is a set of command line tools. They i iextract a database to the file, load this database and run query and then exit. For CodeQL, it works the same way. The user needs to create a database in form of file and then scans it using a command line tool. The solution is similar for VSCode. It extracts database file and then a user can run queries against it.

[0076] However, the CodeFuse solution can work only on a single machine, if it is run in the cloud, it doesn’t have the ability to scale. The same for CodeQL, it is not scalable and it’s not based on CPG and don’t support incremental updates.

[0077] In view of the above, in order to improve the scalability of the whole solution, in the present disclosure, a microservice architecture is proposed, in which a distributed set of scalable components for loading and processing a big CPG in a graph database is introduced and online DSL queries to this database are supported. The main idea is to split the existing solution for analyzing the source code into smaller semi-independent services, which can process parts of the source code in parallel, so they can independently scale horizontally and use only needed resources. This includes the following:

[0078] 1. Extract file processing services and divide them into 3 parts1-a. Source load service / microservice - for loading source code to cloud storage;1 -b. Sharding service / microservice - analyzing and splitting the source code into smaller parts which can be processed in parallel, for the sake of load balance of the payload;1-c. Parser service / microservice (Language specific AST parser) - separate process which can use native language specific parser and transform AST to normalized intermediate form, e.g., unified AST.]0079] 2. Extract CPG build services and divide them into 2 parts2-a. (Adaptive) AST loader service / microservice (or AST Loader for short) - for initial loading unified AST to the database;2-b. CPG resolver service / microservice (CPG Resolver for short) - a set of resolvers which resolves additional edges and properties for static analysis.

[0080] 3. Extract database service / microservice with API for running graph queries and updates.

[0081] 4. Extract DSL Query service / microservice to run analysis algorithms based on DSL query from a user (an end user).

[0082] The splitting of the whole service into separate microservices (the microservice architecture) makes it possible to scale these microservices independently of each other, accommodating for the size of a particular analysis task.

[0083] The proposed architecture can work on multiple machines (or pods speaking in Kubemetes language), thus providing a possibility to use different machines for the same analysis in parallel, thereby solving the scalability problem

[0084] Besides, the split of the existing solution into the above described microservice architecture may be beneficial for improving scalability, resource utilization and latency. Different services can be scaled according to the workload, thereby achieving load balancing. In addition, based on the solution proposed in the present disclosure, a real time database can be obtained and analysis results updates can be achieved, having in memory service it is possible to fit into a target latency using flexible service with real time scalability. Further, it is also proposed to separate the generation / building of the CPG database with the DSL query, such decoupling of different stages of processing improves stability and maintainability.

[0085] For example, if the workload consists of several separate steps, A, B, C and so on, take A and B as an example, different computing efforts may be required to process the workloads of A and B, respectively. For example, A may require M computing nodes, whereas B will require N, with M is not equal to N. So, it’s insufficient to just be able to arbitrarily split the workload to multiple “fragments”, therefore, in the present disclosure, what computing resources are needed for each step of the pipeline (A, B, C and so on). The latency of the whole service or application (as perceived by end users) will highly depend on the ability to decide how many computing resources are available at each step. The embodiments of the present disclosure provide solutions which enable the configuration of the amount of those resources, this would be achieved with the proposed microservice architecture instead of a monolith application. If the computing resources (nodes) are properly shared among microservices, this can ensure minimal (near-real-time) latency of the updates, both to the database and the results of the code analysis.

[0086] Before detailing the solution of the present disclosure, the scenarios in which the embodiments of the present disclosure may be implemented will be described. Generally, the proposed solution may be implemented as a service (or called CPG service or cloud service or cloud-based service) to generate / build a CPG for a source code in a CPG database and make DSL query based on the CPG. The proposed service being a cloud-based service means that the provision of this service would be presented as a series of microservices (including the generation / building of CPG(s) in a CPG database and / or the DSL query based on the CPG database) running at the cloud, e.g., in a cluster. When implementing the service as a cloud-based service, the source code may be uploaded to the cloud for further processing. The proposed service can be in form of cloud-based service with API of loading and DSL queries, and in form of IDE plugin which works with corresponding service. There may be several ways for accessing theproposed service, the accessing means to upload the project to the cloud and to carry out DSL query based on the CPG database which stores CPG(s) corresponding to the project. For example, a user (an end user) may use a web user interface (UI) to access the service, the user may also use an IDE plugin to access the service. In the following, different implementations of accessing the service will be illustrated with reference to drawings.

[0087] Scenario 1: accessing the cloud-based service with a web user interface

[0088] As shown in FIG. 1 A, in this scenario, a user may have some cloud-based services with web interface and public API through which he can set different parameters of his analysis task. And the provision of the service may be guided by the following process:

[0089] 1) Uploading of a source code of a project to a CPG database

[0090] In this step, the user may enter a project name and a link for downloading the project by the cloud-based service. The project name may uniquely identify the source code of the project in the database so that the user can update the project later based on the project name. The link may be a Git link or a patch link, the Git link may be a link to git version control system from which source code can be downloaded, and the patch link may be a link to patch of the project, i.e., source code changes in git patch format. It can be a file path to patch file or some commits range in git.

[0091] Here the CPG database may be a graph database storing multiple CPGs corresponding to multiple projects, and these projects may be from one user or multiple users, which is not limited in the embodiments of the present disclosure.

[0092] 2) Run the project or patch loading (submitting the project for being processed) which may take some time depending on project and amount of allocated resources in the cloud-based service per particular user.

[0093] From the perspective of the cloud-based service, this step would be downloading the source code of the project based on the Git link / patch link.

[0094] 3) After the loading of the project finishes, a notification may be sent to the user through API callback, which can be reflected in graphical user interface (GUI) as a notification window, and further actions may be enabled.

[0095] 4) Run DSL Query

[0096] For a given project, the CPG corresponding to the project will be established in the CPG database based on the above steps 1) to 3), and then the user can open a query editor and put a DSL Query into an edit box. Then the user may press execute and service will return a result in a predefined format, e.g., an SARIF format, which can be visualized in GUI also.

[0097] Scenario 2: accessing the cloud-based service with an Integrated Development Environment (IDE) Plugin

[0098] As shown in FIG. IB, in this scenario, the IDE plugin can access the proposed service through API. When a user opens his project for the first time from Git, it calls API to start building CPG from scratch. Or if the project was already loaded into the CPG database, the CPG service for loading sources will check difference between loaded revision and current revision and update the database, or if changes were not committed upload the difference patch.

[0099] In separate Tab, the user will see CPG Service window where the user can see status, if it is loading then part of the options are disabled, if it is finish loading, then the user can see DSL Query window with DSL editor.

[0100] After DSL Execution the user can see DSL query result in form of subgraph, in which the user can investigate node properties and relations. Also the user can navigate the code, by clicking on to the node or edge, the user can see related pieces of code in the code editor window (shown by the tree-like pattern on the lower right comer), and perform different actions upon them. The tree-like pattern on the lower right comer is a node representation of the project, and contains multiple nodes, each of the multiple nodes would have corresponding text shown right beside the node.

[0101] As shown in FIG. 2, which is a schematic illustration of an exemplary microservice architecture according to one or more embodiments of the present disclosure. Based on the proposed microservice architecture, the above-mentioned scenarios of accessing the cloud-based service are illustrated, including the Web UI and the IDE. The creation of the CPG in the CPG database is implemented based on the source load service, the sharding service, the AST parser service, the adaptive AST loader service, the CPG resolver service. Specifically, the source code being analyzed may be presented as a project model loaded into the memory, or as just a set of source code files residing on a hard disk, so during the creation of the CPG for a project, the source load service may access the file storage to get the source code, then the sharding service splits a large project into smaller fragments (which may also be stored on-disk or in a cloud-based storage), and the AST parser service parses the fragments to generate unified ASTs, then the adaptive AST loader service loads the unified ASTs into the CPG database as CPG, and the CPG resolver service may update the CPG. It should be noted that a series of blocks for the same service refer to multiple instances of the service.

[0102] The CPG database is running as an independent service, so the user may execute DSL query to the CPG database by using the IDE or Web UI, at this point, the DSL query service would access the CPG database to get aquery result.

[0103] The previous description may also be presented in the form of a table, the whole application can be split into smaller semi-independent services running as separate processes, resulting in a microservice architecture which is much more scalable. The services are shown in Table 1.Table 1

[0104] Depending on the size and complexity of the source code being analyzed, each of the above service types can scale (i.e., have multiple identical running instances) independently of the up-stream and down-stream services, resulting in better resource allocation.

[0105] The technical solution provided by the embodiments of the present disclosure suggest a microservice architecture which provides scalable and resilient system for loading and updating CPG database and provides online query for this database.

[0106] The proposed technical solution provides a sequence of components (microservices) for building a CPGfrom source code and loading the CPG into a graph database (each of them can be scaled horizontally), and describes a concrete method of parallelization of CPG creation and updates. Having loading stages as separate services implements single responsibility principle which allows to achieve better scalability and more optimal resource usage, speedup development of new features and language frontends because of less coupling between components. Besides, simultaneous processing of several independent projects may also be optimized by sharing services. Further, sharing of preloaded and analyzed dependencies between projects becomes possible, since a user doesn’t need to load all the dependencies for each project. It is possible to have shared database space with preloaded CPG database for needed versions of dependencies.

[0107] In addition, the CPG graph database is running as a separate microservice, the DSL query request (request service) is independent from CPG creation and modifications, having CPG graph database as a separate service makes independent DSL queries and CPG loading. In this way, a user can load CPG once and then execute DSL queries at any moment, which creates better user experience in development environment. Further, the DSL query request(s) from one or more clients may be handled separately, and the DSL query result may be returned in a format of SARIF, which facilitates further processing by providing a standardized representation of static analysis results, enabling easy integration with various development tools and improving interoperability across different systems.

[0108] In the proposed technical solution, the numbers of parallel instances of different microservices and the database space usage per client are configurable, which makes it possible for multiple users to work with the same project simultaneously, without creating the same database for each user. Besides, it allows to save money to user providing flexible resource management on demand. Further, it allows to create better user experience by fixing latency allowing to reach real time analysis as you type in IDE for example and make more predictable service behavior.

[0109] The above describes possible scenarios or generalized description of the embodiments of the present disclosure, embodiments of the present disclosure are illustrated in the following.

[0110] FIG. 3 shows a flowchart of a data processing method according to one or more embodiments of the present disclosure. The method may be implemented by a device, such as a server. The method may include the following steps.

[0111] S302, obtaining, based on a first number of instances of a first cloud-based microservice, a plurality of unified ASTs corresponding to a plurality of fragments of a target code.

[0112] S3O4, loading, based on a second number of instances of a second cloud-based microservice, the plurality of unified ASTs into a graph database as a CPG of the target code.

[0113] When processing a target code to obtain its corresponding CPG, the plurality of fragments of the target code would be parsed based on a first number of instances of a first cloud-based microservice to obtain a plurality of unified ASTs, and then the plurality of unified ASTs are loaded into a graph database as the CPG of the target code.

[0114] Here the target code may be, e.g., a source code of a project, the project may be a new project which has never been processed, that is, there is no CPG in the graph database corresponding to the project, in this case, the CPG may also be an initial CPG of the target code, and the above steps S302-S304 would be a process of generating the initial CPG for the project. It is also possible that the initial CPG has already been generated for the project, and the target code obtained is an update of the project, then the above steps S302-S304 would be a process of updating the CPG for the project, so in this case, the loading of the plurality of unified ASTs into the graph database as the CPG of the target code means to update the already generated / built CPG of the target code. The CPG may also be referred to as an instance of a CPG graph, it may contain multiple unified ASTs, and may also contain some additional sub-graphs such as control-flow (CFG) graphs.

[0115] The graph database here may also be referred to as a CPG database, it may be a database which stores multiple CPGs corresponding to multiple projects (may be from one end user or different end users). In the proposed solution, CPG graph database is running as a cloud-based microservice independent from the first / second / third cloud-based microservice, the DSL query request (request service) is independent from CPG creation and modifications (which would be accomplished based on the first / second / third cloud-based microservice), having CPG graph database as a separate service makes independent DSL queries and CPG loading. In this way, a user can load CPG once and then execute DSL queries at any moment, which creates better user experience in development environment. In a possible implementation of the present disclosure, one project may correspond to one single CPG. In another possible implementation of the present disclosure, the large project-wide CPG may also be split into smaller sub-graphs for the purposes of parallel processing, memory conservancy, and so on, the number of subgraphs obtained from splitting of a CPG is not limited in the embodiments of the present disclosure. Sometimes the CPG database may have multiple branches reflecting the branches or source code in version control system. The CPG branch may simply contain increment to the some base CPG database snapshot. Also the CPG database may have some shared space for preloaded versions of dependencies which can be referenced from user space.

[0116] The first cloud-based microservice and the second cloud-based microservice are cloud-based services, since they provide their respective microservices at the cloud. The first cloud-based microservice and the second cloud-based microservice are two separate / independent microservices, which means configuration of instances of these two microservices can be done separately. In this cloud service scenario, the target code may be uploaded to the cloud, where fragments of the target code may be processed by the first cloud-based microservice, and unified ASTs obtained by the first cloud-based microservice may be then processed by the second cloud-based microservice, and finally the CPG of the target code may be obtained.

[0117] In a possible implementation of the present disclosure, the first cloud-based microservice may be a parser service, it serves to parse fragments of a target code into corresponding unified ASTs. When implementing the parsing, one or more instances of the first cloud-based microservice may be triggered, that is, the first number of instances of the first cloud-based microservice would be one or more. In a possible implementation of the present disclosure, the number of fragments (or referred to as shards) corresponds to the first number of instances of the first cloud-based microservice that consume them, so a 1 :1 correspondence is maintained in this case. In another possible implementation of the present disclosure, an instance of the first cloud-based microservice may consume two or more shards. The first number of instances of the first cloud-based microservice may get terminated abnormally, and different “shards” may take different (non-equal) time to get processed, which is not limited in the embodiments of the present disclosure. Besides, in a possible implementation of the present disclosure, when there are multiple fragments, they could be distributed among several instances of the parser service, and each parser service may be responsible for one or more fragments, as a result of the parsing, the number of the unified ASTs obtained by the instances of the parser service may equal to the number of fragments, that is, one fragment may lead to one unified AST.

[0118] In a possible implementation of the present disclosure, the second cloud-based microservice may be an AST loader service, it serves to load the plurality of unified ASTs into the graph database as the CPG. When implementing the loading, one or more instances of the second cloud-based microservice may be triggered, that is, the second number of instances of the second cloud-based microservice would be one or more. In a possible implementation of the present disclosure, instances of the second cloud-based microservice are produced on a perfile basis, which means there’s a 1 : 1 correspondence between instances and unified ASTs, basically because this is the way language parsers work: during analysis, compilation, or interpretation, they consume 1 file at a time. As described above, sometimes a single instance of a CPG graph gets usually built, and it corresponds to the wholeproject. The CPG may contain multiple unified ASTs, and also some additional sub-graphs such as control-flow (CFG) graphs.

[0119] The first number and the second number may be numbers of instances of the first and second cloudbased microservices respectively. Either the first number or the second number is configurable. That is, the first number may be configurable based on a workload requirement corresponding to the first cloud-based microservice, the second number may also be configurable based on a workload requirement corresponding to the second cloudbased microservice. It is not required to always configure the first and second numbers, but the configuration of these two numbers may be left as a choice for the end user, so that they can be flexibly set according to the workload of the parsing and the loading operations. In order to improve scalability, resource consumption and / or latency, for example, the workload requirement corresponding to the first cloud-based microservice may be related to the workload (e.g., the number of fragments of the target code) to be handled by the first cloud-based microservice, and the workload requirement corresponding to the second cloud-based microservice may be related to the workload (e.g., the number of unified ASTs) to be handled by the second cloud-based microservice. For another example, the workload requirement corresponding to the first cloud-based microservice and the workload requirement corresponding to the second cloud-based microservice may be hard limits set by the cloud for each end user, each end user may have his corresponding quota, so that the end user can configure the first number and / or second number within his quota for handling his project. The workload requirement corresponding to the first cloud-based microservice may be as same as or different from the workload requirement corresponding to the second cloud-based microservice, which is not limited in the embodiments of the present disclosure. It should be noted that different users / clients of the same set of microservices may have different resource quotas. Here a client may be a device manipulated by a user to submit his project, configure parameters (e.g., first / second / third number) for processing the project, etc.]0120] The separation of the first cloud-based microservice and the second cloud-based microservice makes it possible to separately configure their respective instances, that is, the first number and the second number are both configurable, separate configuration for the instances of the two cloud-based microservices may be beneficial for improving scalability, resource utilization and latency. Since the first number and the second number are configurable, so the computing resources (nodes) are properly shared among microservices, this can ensure minimal (near-real-time) latency of the updates, both to the database and the results of the code analysis.

[0121] In a possible implementation of the present disclosure, the configuration of the first number and / or thesecond number is performed locally at a client from which the target code is submitted. In this way, the client (end user) may locally configure the first number and / or the second number, and submit the target code to the cloud, so that the cloud could deal with the target code based on the first number and / or the second number configured by the client, thereby allowing for customization and optimization of the development process according to the client’s specific needs and environment, as well as saving computing resource of the client.

[0122] In a possible implementation of the present disclosure, the configuration of the first number and / or the second number is based on a size of the target code. In a possible implementation of the present disclosure, the configuration of the third number is also based on a size of the target code. When the target code occupies a large space, it would be more reasonable to allocate more computing resources for handling this target code, so more instances for building the CPG of the target code (i.e., the first and second numbers) and more instances for updating the CPG may be configured, thereby improving the adaptivity of the whole service for handing different sizes of target codes.

[0123] In a possible implementation of the present disclosure, after step S204, the method may further include:updating, based on a third number of instances of a third cloud-based microservice, the CPG of the target code.

[0124] In some cases, the CPG graph, as built from the unified ASTs, may be missing certain information (such as missing nodes and / or edges) or, alternatively, have certain information in the de-normalized state. Further, invocations of the same C++ or Java function may produce different “function call” nodes which would contain references to the different (but semantically identical) “function declaration” nodes. Therefore, in these cases, it is possible to call the third cloud-based microservice to update the CPG.

[0125] Similar as the first and second cloud-based microservices, the third cloud-based microservice is also a cloud-based services, since it provides its microservice at the cloud. The third cloud-based microservice may be called a CPG resolver service, it would be the responsibility of the CPG resolver service to intelligently analyze such edges and vertices in a graph and perform de-duplication.

[0126] When implementing the updating, one or more instances of the third cloud-based microservice may be triggered, that is, the third number of instances of the third cloud-based microservice would be one or more. There is no limitation on the number of extra edges, the number of nodes, or the necessity of excluding de-duplication on the CPG to be handled by one instance of the third cloud-based microservice, which could be set according to actual needs.

[0127] The updating of the CPG of the target code may include one or more of: adding at least one extra edge or graph for the CPG of the target code, the at least one extra edge or graph may include, e.g., call edge, refersTo edges, one or more data flow graphs (DFG), one or more code flow graphs (CFG); updating at least one property of at least one node in the CPG of the target code, for example, the at least one property of the at least one node may be a de-normalized state of each of the at least one node; or performing de-duplication on the CPG of the target code, for example, two invocations of the same C++ or Java function may produce two “function call” nodes which would contain references to the two different (but semantically identical) “function declaration” nodes, in this case, it is possible to call the third cloud-based microservice to update the CPG to remove the duplication. Various ways of updating are provided, so that the flexibility for updating the CPG is improved and the accuracy of the updated CPG is increased.

[0128] In a possible implementation of the present disclosure, the third number is configurable based on a workload requirement corresponding to the third cloud-based microservice.

[0129] Similar to the parser service (the first cloud-based microservice) and the loader service (the second cloud-based microservice), the third cloud-based microservice may also be configurable, it is not required to always configure the third number, but the configuration of the third number maybe left as a choice for the end user, so that it can be flexibly set according to the workload of the updating operation. In order to improve resource consumption and / or latency, for example, the workload requirement corresponding to the third cloud-based microservice maybe related to the workload (e.g., the number of extra edges, the number of nodes, the necessity of excluding deduplication on the CPG) to be handled by the third cloud-based microservice. For another example, the workload requirement corresponding to the third cloud-based microservice may be a hard limit set by the cloud for each end user, each end user may have his corresponding quota, so that the end user can configure the third number within his quota for handling his project. The workload requirement corresponding to the third cloud-based microservice may be same as or different from the workload requirement corresponding to the first / second cloud-based microservice, which is not limited in the embodiments of the present disclosure.

[0130] As described above, the number of instances for the first, second and third cloud-based microservices may all be configurable for the sake of improving the scalability, the resource utilization and the latency. The configuration of these cloud-based microservices may be based on their respective workload requirement, and the workload requirement may be implemented as follows.

[0131] In a possible implementation of the present disclosure, the workload requirement corresponding to thefirst cloud-based microservice includes a maximum number of instances allocable for a client from which the target code is submitted (which would be regarded as a hard limit for the client) and used for running the first cloud-based microservice. The client may be a device used by a user for submitting the target code. Here the maximum number of instances allocable for the client generally sets an upper boundary of the instances which can be allocated to the client, which is not necessarily be the real instances allocated to the client. So when configuring the first number based on the workload requirement corresponding to the first cloud-based microservice, this first number would be smaller or equal to the maximum number of instances allocable for the client, e.g., depending on the number of fragments of the target code to be handled by the first cloud-based microservice.

[0132] In a possible implementation of the present disclosure, the workload requirement corresponding to the second cloud-based microservice includes a maximum number of instances allocable for a client from which the target code is submitted (which would be regarded as a hard limit for the client) and used for running the second cloud-based microservice. The client may be a device used by a user for submitting the target code. Here the maximum number of instances allocable for the client generally sets an upper boundary of the instances which can be allocated to the client, which is not necessarily be the real instances allocated to the client. So when configuring the second number based on the workload requirement corresponding to the second cloud-based microservice, this second number would be smaller or equal to the maximum number of instances allocable for the client, e.g., depending on the number of unified ASTs to be handled by the second cloud-based microservice.

[0133] In a possible implementation of the present disclosure, the workload requirement corresponding to the third cloud-based microservice includes a maximum number of instances allocable / allocated for a client from which the target code is submitted (which would be regarded as a hard limit for the client) and used for running the third cloud-based microservice. The client may be a device used by a user for submitting the target code. Here the maximum number of instances allocable for the client generally sets an upper boundary of the instances which can be allocated to the client, which is not necessarily be the real instances allocated to the client. So when configuring the third number based on the workload requirement corresponding to the third cloud-based microservice, this third number would be smaller or equal to the maximum number of instances allocable for the client, e.g., depending on the number of extra edges, the number of nodes, the necessity of excluding de-duplication on the CPG to be handled by the third cloud-based microservice.

[0134] In addition to the configuration of the first, second and third cloud-based microservices, the solution of the present disclosure also proposes several configurable settings with respect to the graph database. In a possibleimplementation of the present disclosure, at least one of following settings is configurable for the graph database: a space usage for the CPG in the graph database; the number of instances used for running nodes of the CPG; or the number of replicas of the graph database allocable for a client from which the target code is submitted, here the client may be a device used by a user for submitting the target code. Here the number of replicas of the graph database allocable for the client generally sets an upper boundary of the replicas which can be allocated to the client, which is not necessarily be the real number of replicas allocated to the client. So when configuring the number of replicas, this number would be smaller or equal to the number of instances allocable for the client. The resource consumed by the CPG can be configured in various ways, allowing for controlled development of the CPG, this flexibility makes the graph database more adaptive, especially when it is used to store CPGs for multiple clients.

[0135] Further, the analysis of the DSL query may also be configured. For example, it may be based on at least one of following settings : a size of computing resources allocated for the analyzing; a size of memories allocated for the analyzing; or, the number of DSL query requests executable within a predefined time period (e.g., minute, hour, day, etc.). By configuring the computing resources, such as CPU and memory, for carrying out DSL analysis, the overall flexibility and efficiency of the system can be significantly improved.space usage per client are configurable, which makes it possible for multiple users to work with the same project simultaneously, without creating the same database for each user. Besides, it allows to save money to user providing flexible resource management on demand. Further, it allows to create better user experience by fixing latency allowing to reach real time analysis as you type in IDE for example and make more predictable service behavior.[01371 FIG. 4 shows a schematic illustration of settings for cloud-based microservices according to one or more embodiments of the present disclosure. As shown in FIG. 4, one of the main features of the proposed cloud-based service is its scalability, several settings may be provided for each user / each client.101381 For example, for the parser service, the loader service and the CPG Resolver service, the maximum number of services, that is, how many instances of services can be allocated per user may be configured. As shown in FIG. 4, for AST parser service, how many AST parser instances can be run simultaneously may be configured, for the Adaptive AST Loader service, how many adaptive AST loader instances can be run simultaneously may be configured, and for the CPG Resolver service, how many CPG Resolver instances can be run simultaneously may be configured.

[0139] For the CPG Database, following CPG graph database settings would be possible: 1) User space - howmany memory and disk space can be allocated per user; 2) Instances - how many database node instances can be run per user, this is for horizontal database scaling; 3) Replicas - how many replicas of CPG graph per user are to allocate, this is for durability and disaster recovery.[0140) For the DSL Analysis service (which refers to the execution of the DSL analysis), following settings would be possible: 1) CPU usage - the number of CPU units which can be allocated for analysis services; 2) Memory usage - how much memory can be used by analysis services; 3) Requests - how many DSL requests can be executed by user per hour.10141] Besides, the loading latency expected by the user may also be configured, e.g., some value per one million lines.

[0142] It should be noted that names of the microservices / settings and deployment of the boxes for different microservices in the UI throughout the text and the drawings are merely illustrative rather than restrictive.

[0143] In a possible implementation of the present disclosure, before the obtaining of the plurality of unified ASTs, the method further includes:downloading the target code based on a link, where the link indicates a location where the target code is stored or a location where a patch of the target code is stored;splitting the target code to obtain the plurality of fragments.

[0144] As described before, the proposed service is a cloud-based service, so it has the ability to download the target code based on a link, this link may be instructed by the client / user. Since the target code may be a source code of a new project, or may be a source code of a project whose CPG has already been built in the graph database, the link for downloading the target code may also indicate a location where the target code is stored or a location where a patch of the target code is stored. When the link indicates a location where a patch of the target code is stored, the cloud would know, based on the link, that the target code has already got its corresponding CPG, so the loader service and the CPG resolver would handle the unified ASTs from the parser service to update the built CPG in the graph database.

[0145] In a possible implementation of the present disclosure, the indication of the location where the target code is stored may be implemented as a Video Cloud Storage (VCS) link, where a user can either have version control (VCS) root information, like a Git or a Mercurial URL, where the server can download the source code. In a possible implementation of the present disclosure, the indication of the location where a patch of the target code is stored may be implemented as a patch link, where a user can have a “snapshot” of his source code, probably evenw / o any VCS information, and a patch (in the form of a context diff or a unified difi), which gets applied to the above source snapshot. This is the natural way how incremental analysis of the source code can be performed.

[0146] The downloading and the splitting of the target code into fragments may also be implemented based on instances of microservices. For example, the downloading may be based on one or more instances of a source load service, and the splitting of the target code into fragments may be based on one or more instances of a sharding service. The source load service and the sharding service are also cloud-based microservices, so the number of instances for either the source load service or the sharding service may also be configurable based on a corresponding workload requirement, similar as the first / second / third cloud-based microservice, the workload requirement corresponding to the source load service may also be related to the workload (e.g., the size of the target code) to be handled by the source load service, and the workload requirement corresponding to the sharding service may also be related to the workload (e.g., the size of the target code) to be handled by the sharding service.|0147] In a possible implementation of the present disclosure, the method may further include:receiving at least one domain specific language (DSL) query request from at least one corresponding client;for each of the at least one DSL query request, analyzing the graph database based on a CPG in the graph database corresponding to the DSL query request to obtain a corresponding DSL query result, where data in the DSL query result is in a static analysis results interchange format.

[0148] In a possible implementation of the present disclosure, the at least one DSL query request comprises multiple DSL query requests, and at least one of the multiple DSL query requests is submitted locally by a corresponding client based on an integrated development environment (IDE) instance at the corresponding client. When there are multiple DSL query requests, one or more of the multiple DSL query requests can be submitted via one or more IDE instances. Submitting the DSL query request locally from an IDE instance would enhance performance, reduce latency by processing requests on the client side, leading to faster response times.

[0149] The above at least one DSL query request may include one or more DSL query requests, and the multiple DSL query requests may be from one user or different clients, which is not limited in the embodiments of the present disclosure. When dealing with each of the at least one DSL query request, analysis is performed based on a CPG in the graph database corresponding to the DSL query request, and the DSL query result may be returned in a format of SARIF, which facilitates further processing by providing a standardized representation of static analysis results, enabling easy integration with various development tools and improving interoperability across different systems.

[0150] In order to illustrate the whole scheme more clearly, in the following, examples which show the process of building a CPG in a CPG database for a source code and querying the CPG database will be described.

[0151] FIG. 5 is a schematic illustration of an exemplary pipeline for provision of a cloud-based microservices according to one or more embodiments of the present disclosure. As shown in FIG. 5, the whole process of analysis of a DSL query for a source code (project) may be separated into two main parts, the upper part of FIG. 5 shows the process of building a CPG for the project, and the lower part of FIG. 5 shows the execution of the DSL query from the user. In the proposed solution, the CPG graph database is running as a separate microservice, and the DSL query request (request service) from the user is independent from CPG creation and modifications, having CPG graph database as a separate service makes independent DSL queries and CPG loading. In this way, a user can load CPG once and then execute DSL queries at any moment, which creates better user experience in development environment.

[0152] Specifically, as shown in the upper part of FIG. 5, based on the proposed microservice architecture, the source code from the user is processed by different services to get a CPG corresponding to the source code in the CPG database. The user may provide the source code in various forms, including incremental updates (patches), then the service proposed by the present disclosure downloads the source code (e.g., by cloning a Git repository) and stores the source code locally (e.g., at the cloud), this part may correspond to the above-mentioned provision of the source load service. Then the sharding service splits a large project into smaller fragments and distributes them between parser service instances, and the parser service instance parses one or more fragments using native tools or non native tools (not limited in the embodiments of the present disclosure), such as Chang, CPython or Node.js, and serializes it to a unified AST form. After the unified ASTs from parser service instances are obtained, the AST loader service gets the unified AST from data streams such as Kafka or reactive HTTP streams and loads it to the graph database. As a preferred implementation, the CPG resolver service may run a set of resolver processes to enrich graph in the CPG database by adding extra edges, such as call edge, refersTo edges, DFG, CFG, update nodes data etc.

[0153] The creation / building of the CPG in the CPG database would be independent to the execution of DSL query. As shown in lower part of FIG. 5, the user may execute a DSL query to the Graph Database (storing one or more CPGs) using a special query language, and then the user can see the results of his query in the form of selected nodes of the CPG, or fragments of the source code, or as SARIF data.

[0154] According to the embodiments of the present disclosure, the plurality of unified ASTs corresponding to a plurality of fragments of a target code are obtained based on a first number of instances of a first cloud-basedPC17RU2024 / 000341microservice, and then the plurality of unified ASTs are loaded into the graph database as a CPG of the target code. The separation of the first cloud-based microservice and the second cloud-based microservice makes it possible to separately configure their respective instances, that is, the first number and the second number are both configurable, separate configuration for the instances of the two cloud-based microservices may be beneficial for improving scalability, resource utilization and latency. Since the first number and the second number are configurable, so the computing resources (nodes) are properly shared among microservices, this can ensure minimal (near-real-time) latency of the updates, both to the database and the results of the code analysis. The proposed microservice architecture can work on multiple machines (or pods speaking in Kubemetes language), thus providing a possibility to use different machines for the same analysis in parallel, thereby solving the scalability problem. Besides, different services can be scaled according to workload, thereby achieving load balancing. In addition, based on the solution proposed in the present disclosure, areal time database can be obtained and analysis results updates can be achieved, having in memory service it is possible to fit into a target latency using flexible service with real time scalability.

[0155] Next, embodiments of products related to the data processing method will be described.

[0156] FIG. 6 shows a schematic structural diagram of a data processing apparatus according to one or more embodiments of the present disclosure. As shown in FIG. 6, the apparatus includes an obtaining module 602 and a loading module 604.

[0157] The obtaining module 602 is configured to obtain, based on a first number of instances of a first cloudbased microservice, a plurality of unified abstract syntax trees (ASTs) corresponding to a plurality of fragments of a target code.

[0158] The loading module 604 is configured to load, based on a second number of instances of a second cloudbased microservice, the plurality of unified ASTs into a graph database as a code property graph (CPG) of the target code, where the first number is configurable based on a workload requirement corresponding to the first cloud-based microservice, and / or, the second number is configurable based on a workload requirement corresponding to the second cloud-based microservice.]0159] In a possible implementation of the present disclosure, the apparatus further includes:an updating module (not shown), configured to update, based on a third number of instances of a third cloud-based microservice, the CPG of the target code;where the updating of the CPG of the target code includes at least one of following operations: adding at least one extra edge or graph for the CPG of the target code;updating at least one property of at least one node in the CPG of the target code; orperforming de-duplication on the CPG of the target code.

[0160] In a possible implementation of the present disclosure, the third number is configurable based on a workload requirement corresponding to the third cloud-based microservice, and the workload requirement corresponding to the third cloud-based microservice includes a maximum number of instances allocable for a client from which the target code is submitted and used for running the third cloud-based microservice .

[0161] In a possible implementation of the present disclosure, the workload requirement corresponding to the first cloud-based microservice includes a maximum number of instances allocable for a client from which the target code is submitted and used for running the first cloud-based microservice.

[0162] In a possible implementation of the present disclosure, the workload requirement corresponding to the second cloud-based microservice includes a maximum number of instances allocable for a client from which the target code is submitted and used for running the second cloud-based microservice.

[0163] In a possible implementation of the present disclosure, configuration of the first number and / or the second number is performed locally at a client from which the target code is submitted.

[0164] In a possible implementation of the present disclosure, at least one of following settings is configurable for the graph database:a space usage for the CPG in the graph database;a number of instances used for running nodes of the CPG; ora number of replicas of the graph database allocable for a client from which the target code is submitted.

[0165] In a possible implementation of the present disclosure, configuration of the first number and / or the second number is based on a size of the target code.

[0166] In a possible implementation of the present disclosure, the obtaining module 602 is further configured to:download the target code based on a link, where the link indicates a location where the target code is stored or a location where a patch of the target code is stored;where the apparatus further includes a splitting module (not shown), configured to:split the target code to obtain the plurality of fragments.

[0167] In a possible implementation of the present disclosure, the apparatus further includes a receiving module (not shown), configured to:receive at least one domain specific language (DSL) query request from at least one corresponding client; where the apparatus further includes an analyzing module (not shown), configured to:for each of the at least one DSL query request, analyze the graph database based on a CPG in the graph database corresponding to the DSL query request to obtain a corresponding DSL query result, where data in the DSL query result is in a static analysis results interchange format (SARIF).|0168] In a possible implementation of the present disclosure, the at least one DSL query request comprises multiple DSL query requests, and at least one of the multiple DSL query requests is submitted locally by a corresponding client based on an integrated development environment (IDE) instance at the corresponding client.

[0169] In a possible implementation of the present disclosure, the analyzing of the graph database is based on at least one of following settings:a size of computing resources allocated for the analyzing;a size of memories allocated for the analyzing; or,a number of DSL query requests executable within a predefined time period.

[0170] In a possible implementation of the present disclosure, the first cloud-based microservice is an AST parser service, and the second cloud-based microservice is an AST loader service.

[0171] It should be noted that, the obtaining module, the loading module, the updating module, the analyzing module, the splitting module and the receiving module are only illustrative for their functions, and in practice, the functions of the obtaining module, the loading module, the updating module, the analyzing module, the splitting module and the receiving module may be implemented by one generating module, which is not limited here. It should also be noted that, the functions of the obtaining module, the loading module, the updating module, the analyzing module, the splitting module and the receiving module may also be implemented by a single module, and the functions of multiple modules may also be implemented by one or more modules, which is not limited in the present disclosure. The apparatus may be in a form of a computing device, an electronic device, etc.

[0172] The data processing apparatus may be implemented as described in the above method embodiments. It should be understood by a person skilled in the art that, the relevant description of the above modules in the embodiments of the present disclosure may be understood with reference to the relevant description of the data processing method in the embodiments of the present disclosure.

[0173] FIG. 7 is a schematic structural diagram of a computing device according to one or more embodiments of the present disclosure. As shown in FIG. 7, the computing device 700 includes a processor 702, an interface 704for communicating with other devices, a memory 706, and a bus 708. The processor 702, the interface 704, and the memory 706 are in communication with each other via the bus 708. The memory 706 may be stored with computer execution instructions, and the processor 702 executes computer execution instructions stored in the memory 706 to enable the computing device to execute any of the above methods. The computing device 700 can be a server or a terminal device. The number of processors and memories in the computing device 700 is not limited in the embodiments of the present disclosure.

[0174] The bus 708 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one line is used in FIG. 7, but it does not mean that there is only one bus or one type of bus. The bus 708 may include pathways for transferring information between various components of the computing device 700 (e.g., the memory 706, the processor 702, or the interface 704).

[0175] The processor 702 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP) processor.

[0176] The memory 706 can include volatile memory, such as random access memory (RAM). The processor 702 can also include non-volatile memory, such as read-only memory (ROM), flash memory, a mechanical hard disk (HDD), or a solid state drive (SSD).|0177] An executable program code is stored in the memory 706, and the processor 702 executes the executable program code to implement the functions of the aforementioned determining module 702 and the refining module 704, respectively, so as to implement the method described in the above embodiments. That is, the memory 706 has instructions on it for executing the method described in the above embodiments.(0178) The structure of the computing device 700 enumerated above is only an exemplary illustration, and the present disclosure is not limited thereto. The computing device 700 of embodiments of the present application includes a variety of hardware found in computer systems in the prior art, e.g., the computing device 700 also includes memory other than the memory 706, such as, for example, a disk memory, and the like. It should be understood by those skilled in the art that the computing device 700 can also include other devices necessary to achieve proper operation. In addition, depending on the specific needs, one of skill in the art should understand that the above-described computing device 700 may also include other hardware devices for realizing other additional functions. In addition, one of skill in the art should understand that the above-described computing device 700 may also include only the devices necessary to realize the embodiments of the present disclosure, and need not includeall of the devices shown in FIG. 7.

[0179] FIG. 8 is a schematic diagram of an architecture of a computing device cluster according to one or more embodiments of the present disclosure. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0180] As shown in FIG. 8, the cluster of computing devices described includes at least one computing device 800. The memory 806 in one or more of the computing devices 800 in the cluster of computing devices can hold the same instructions for performing the method described in the above embodiments.

[0181] In some possible implementations, some of the instructions for performing the method described in the above embodiments can also be separately held in the memory 806 of the one or more computing devices 800 in the cluster of computing devices. In other words, a combination of the one or more computing devices 800 can jointly execute instructions for performing the method described in the above embodiments.

[0182] It is noted that the memories 806 in the different computing devices 800 in the cluster of computing devices can store different instructions for performing some of the functions of the computing devices 800, respectively. That is, the instructions stored in the memories 806 in the different computing devices 800 can realize the functions of one or more of the modules described with reference to FIG. 6.

[0183] In some possible implementations, one or more computing devices in a cluster of computing devices can be connected via a network. Among other things, the network can be a wide area network or a local area network, etc. FIG. 9 illustrates one possible implementation. FIG. 9 is a schematic diagram of a connection between computing devices 900A and 900B over a network according to one or more embodiments of the present disclosure. The two computing devices 900A and 900B are connected to each other via a network. Specifically, the connection to said network is made through a communication interface in each computing device. In this class of possible implementations, the memory 906A in the computing device 900A holds instructions for performing the functions of the obtaining module 602. At the same time, the memory 906B in the computing device 900B holds instructions for performing the functions of the loading module 604. Other modules described with reference to FIG. 6 may also be implemented with computing devices as 900A and 900B, which will not be illustrated for the sake of brevity.]0184] The connection between the clusters of computing devices shown in FIG. 9 can be such that, considering that the method provided in the present disclosure requires processing a large number of data, it is contemplated thatthe functions realized by the obtaining module 602 be performed by the computing device 900B.|0185] The functions of computing device 900A illustrated in FIG. 9 can also be accomplished by multiple computing devices. Similarly, the functions of computing device 900B can be accomplished by multiple computing devices.

[0186] In some aspects of the present disclosure, there is provided a data processing apparatus including processing circuitry for executing any of the above methods. It should be understood that the apparatus can execute the steps in the above method embodiments, which will not be repeated here.

[0187] In some aspects of the present disclosure, there is provided a computing device cluster including at least one computing device, and the computing device includes a processor and a memory coupled with the processor. The memory is storing instructions which, when executed by the processor, cause the computing device cluster to perform any of the above methods.

[0188] In some aspects of the present disclosure, there is provided a chip, including an input / output (I / O) interface and a processor, where the processor is configured to call and run computer execution instructions stored in a memory, to enable a device installing with the chip to execute any of the above methods.

[0189] In some aspects of the present disclosure, there is provided a computer-readable medium storing computer execution instructions which, when run on a computing device cluster, cause the computing device cluster to execute any of the above methods.

[0190] In some aspects of the present disclosure, there is provided a computer program product including computer execution instructions which, when run on a computing device cluster, cause the computing device cluster to execute any of the above methods.

[0191] In some aspects of the present disclosure, there is provided a computer program including computer execution instructions which, when run on a computing device cluster, cause the computing device cluster to execute any of the above methods.

[0192] The term “include” used herein and its variations are open inclusion, that is, “include but not limited to”. The term “based on” means “at least partly based on”. The term “an embodiment” represents “at least one embodiment”; the term “another embodiment” represents “at least one another embodiment”; and the term “some embodiments” represents “at least some embodiments”. Related definitions of other terms will be provided in the following.

[0193] It should be noted that concepts such as “first”, and “second” mentioned in the present disclosure aremerely used to distinguish different apparatuses, modules, or units, but not to limit the sequence or interdependency of functions executed by these apparatuses, modules, or units.

[0194] It should be noted that the singular or plural modification mentioned in the present disclosure is illustrative and not restrictive, and those skilled in the art should understand that it should be understood as “one or more” unless clearly defined in the context otherwise.

[0195] Although the present disclosure describes methods and processes with steps in a certain order, one or more steps of the methods and processes may be omitted or altered as appropriate. One or more steps may take place in an order other than that in which they are described, as appropriate.

[0196] Note that the expression “at least one of A or B”, as used herein, is interchangeable with the expression “A and / or B”. It refers to a list in which you may select A or B or both A and B. Similarly, “at least one of A, B, or C”, as used herein, is interchangeable with “A and / or B and / or C” or “A, B, and / or C”. It refers to a list in which you may select: A or B or C, or both A and B, or both A and C, or both B and C, or all of A, B and C. The same principle applies for longer lists having a same format.

[0197] Although the present disclosure is described, at least in part, in terms of methods, a person of ordinary skill in the art will understand that the present disclosure is also directed to the various components for performing at least some of the aspects and features of the described methods, be it by way of hardware components, software or any combination of the two. Accordingly, the technical solution of the present disclosure may be embodied in the form of a software product. A suitable software product may be stored in a pre-recorded storage device or other similar non-volatile or non-transitory computer readable medium, including DVDs, CD-ROMs, USB flash disk, a removable hard disk, or other storage media, for example. The software product includes instructions tangibly stored thereon that enable a processing device (e.g., a personal computer, a server, or a network device) to execute examples of the methods disclosed herein. The machine-executable instructions may be in the form of code sequences, configuration information, or other data, which, when executed, cause a machine (e.g., a processor or other processing device) to perform steps in a method according to examples of the present disclosure.

[0198] The present disclosure may be embodied in other specific forms without departing from the subject matter of the claims. The described example embodiments are to be considered in all respects as being only illustrative and not restrictive. Selected features from one or more of the above-described embodiments may be combined to create alternative embodiments not explicitly described, features suitable for such combinations being understood within the scope of this disclosure.

[0199] All values and sub-ranges within disclosed ranges are also disclosed. Also, although the systems, devices and processes disclosed and shown herein may include a specific number of elements / components, the systems, devices and assemblies could be modified to include additional or fewer of such elements / components. For example, although any of the elements / components disclosed may be referenced as being singular, the embodiments disclosed herein could be modified to include a plurality of such elements / components. The subject matter described herein intends to cover and embrace all suitable changes in technology.

[0200] Although embodiments have been described above with reference to the accompanying drawings, those of skill in the art will appreciate that variations and modifications may be made without departing from the scope thereof as defined by the appended claims.

Claims

CLAIMS1. A data processing method, comprising:obtaining, based on a first number of instances of a first cloud-based microservice, a plurality of unified abstract syntax trees (ASTs) corresponding to a plurality of fragments of a target code; andloading, based on a second number of instances of a second cloud-based microservice, the plurality of unified ASTs into a graph database as a code property graph (CPG) of the target code;wherein the first number is configurable based on a workload requirement corresponding to the first cloudbased microservice, and / or, the second number is configurable based on a workload requirement corresponding to the second cloud-based microservice.

2. The method according to claim 1, further comprising:updating, based on a third number of instances of a third cloud-based microservice, the CPG of the target code; wherein the updating of the CPG of the target code comprises at least one of following operations: adding at least one extra edge or graph for the CPG of the target code;updating at least one property of at least one node in the CPG of the target code; orperforming de-duplication on the CPG of the target code.

3. The method according to claim 2, wherein the third number is configurable based on a workload requirement corresponding to the third cloud-based microservice, and the workload requirement corresponding to the third cloudbased microservice comprises a maximum number of instances allocable for a client from which the target code is submitted and used for running the third cloud-based microservice.

4. The method according to any one of claims 1 to 3, wherein the workload requirement corresponding to the first cloud-based microservice comprises a maximum number of instances allocable for a client from which the target code is submitted and used for running the first cloud-based microservice.

5. The method according to any one of claims 1 to 4, wherein the workload requirement corresponding to the second cloud-based microservice comprises a maximum number of instances allocable for a client from which the target code is submitted and used for running the second cloud-based microservice.

6. The method according to any one of claims 1 to 5, wherein configuration of the first number and / or the second number is performed locally at a client from which the target code is submitted.for the graph database:a space usage for the CPG in the graph database;a number of instances used for running nodes of the CPG; ora number of replicas of the graph database allocable for a client from which the target code is submitted.

8. The method according to any one of claims 1 to 7, wherein configuration of the first number and / or the second number is based on a size of the target code.

9. The method according to any one of claims 1 to 8, wherein before the obtaining of the plurality of unified ASTs, the method further comprises:downloading the target code based on a link, wherein the link indicates a location where the target code is stored or a location where a patch of the target code is stored;splitting the target code to obtain the plurality of fragments.

10. The method according to any one of claims 1 to 9, further comprising:receiving at least one domain specific language (DSL) query request from at least one corresponding client; for each of the at least one DSL query request, analyzing the graph database based on a CPG in the graph database corresponding to the DSL query request to obtain a corresponding DSL query result, wherein data in the DSL query result is in a static analysis results interchange format (SARIF).

11. The method according to claim 10, wherein the at least one DSL query request comprises multiple DSL query requests, and at least one of the multiple DSL query requests is submitted locally by a corresponding client based on an integrated development environment (IDE) instance at the corresponding client.

12. The method according to claim 10 or 11, wherein the analyzing of the graph database is based on at least one of following settings:a size of computing resources allocated for the analyzing;a size of memories allocated for the analyzing; or,a number of DSL query requests executable within a predefined time period.

13. The method according to any one of claims 1 to 12, wherein the first cloud-based microservice is an AST parser service, and the second cloud-based microservice is an AST loader service.

14. A data processing apparatus, comprising:an obtaining module, configured to obtain, based on a first number of instances of a first cloud-based microservice, a plurality of unified abstract syntax trees (ASTs) corresponding to a plurality of fragments of a targetcode; anda loading module, configured to load, based on a second number of instances of a second cloud-based microservice, the plurality of unified ASTs into a graph database as a code property graph (CPG) of the target code;wherein the first number is configurable based on a workload requirement corresponding to the first cloudbased microservice, and / or, the second number is configurable based on a workload requirement corresponding to the second cloud-based microservice.

15. The apparatus according to claim 14, further comprising:an updating module, configured to update, based on a third number of instances of a third cloud-based microservice, the CPG of the target code;wherein the updating of the CPG of the target code comprises at least one of following operations: adding at least one extra edge or graph for the CPG of the target code;updating at least one property of at least one node in the CPG of the target code; orperforming de-duplication on the CPG of the target code.

16. The apparatus according to claim 15, wherein the third number is configurable based on a workload requirement corresponding to the third cloud-based microservice, and the workload requirement corresponding to the third cloud-based microservice comprises a maximum number of instances allocable for a client from which the target code is submitted and used for running the third cloud-based microservice.

17. The apparatus according to any one of claims 14 to 16, wherein the workload requirement corresponding to the first cloud-based microservice comprises a maximum number of instances allocable for a client from which the target code is submitted and used for running the first cloud-based microservice.

18. The apparatus according to any one of claims 14 to 17, wherein the workload requirement corresponding to the second cloud-based microservice comprises a maximum number of instances allocable for a client from which the target code is submitted and used for running the second cloud-based microservice.

19. The apparatus according to any one of claims 14 to 18, wherein configuration of the first number and / or the second number is performed locally at a client from which the target code is submitted.

20. The apparatus according to any one of claims 1 to 19, wherein at least one of following settings is configurable for the graph database:a space usage for the CPG in the graph database;a number of instances used for running nodes of the CPG; ora number of replicas of the graph database allocable for a client from which the target code is submitted.

21. The apparatus according to any one of claims 1 to 20, wherein configuration of the first number and / or the second number is based on a size of the target code.

22. The apparatus according to any one of claims 1 to 21, wherein the obtaining module is further configured to:download the target code based on a link, wherein the link indicates a location where the target code is stored or a location where a patch of the target code is stored;wherein the apparatus further comprises a splitting module, configured to:split the target code to obtain the plurality of fragments.

23. The apparatus according to any one of claims 1 to 22, further comprising a receiving module, configured to:receive at least one domain specific language (DSL) query request from at least one corresponding client; wherein the apparatus further comprises an analyzing module, configured to:for each of the at least one DSL query request, analyze the graph database based on a CPG in the graph database corresponding to the DSL query request to obtain a corresponding DSL query result, wherein data in the DSL query result is in a static analysis results interchange format (SARIF).

24. The apparatus according to claim 23, wherein the at least one DSL query request comprises multiple DSL query requests, and at least one of the multiple DSL query requests is submitted locally by a corresponding client based on an integrated development environment (IDE) instance at the corresponding client.

25. The apparatus according to claim 23 or 24, wherein the analyzing of the graph database is based on at least one of following settings:a size of computing resources allocated for the analyzing;a size of memories allocated for the analyzing; or,a number of DSL query requests executable within a predefined time period.

26. The apparatus according to any one of claims 1 to 25, wherein the first cloud-based microservice is an AST parser service, and the second cloud-based microservice is an AST loader service.

27. A computing device cluster, comprising at least one computing device, wherein the computing device comprises at least one processor and a memory coupled with the at least one processor, wherein the memory stores instructions which, when executed by the at least one processor, cause the computing device cluster to execute themethod according to any one of claims 1 to 13.

28. A computer-readable medium storing computer execution instructions which, when run on a computing device cluster, cause the computing device cluster to execute the method according to any one of claims 1 to 13.

29. A computer program product comprising computer execution instructions which, when run on a computing device cluster, cause the computing device cluster to execute the method according to any one of claims 1 to 13.

Images