Online machine learning big data job algorithm planner

Abstract

A big data job planner uses online machine learning to recommend algorithms for each job in a sequence to enhance various aspects of job execution. A job algorithm prediction server processes a job's pre-runtime metadata to create predictions of relevant big data job behavior characteristics based on previously learned models of the corresponding big data system's behavior and provides a recommended algorithm to the client big data system. After the big data job is executed, the job metadata coordinator persists pre-runtime metadata and corresponding post-runtime big data job behavioral metadata actually observed. Repeatedly through time, online machine learning is performed on this sequence of job behavior metadata so that machine learning models will learn difficult-to- recognize phenomena, as well as, changes in system behavior through time. Unlike statically coded rules, the system models changing patterns and behaviors, enabling the algorithm predictions to remain useful, thereby enhancing big data system quality.

Description

Attorney Docket No. RAMP-00306-WOONLINE MACHINE LEARNING BIG DATA JOB ALGORITHM PLANNERBACKGROUND OF THE INVENTION

[0001] Big data jobs, particularly those serving production systems instead of one-off analytics queries, are often extremely large and run at a high frequency. Such jobs are designed with many tradeoffs in mind. Even for a single system or two closely related systems, there can be variations in input characteristics, variations in infrastructure health, and other interrelated, dynamic phenomena. All of these various criteria can mean that the previously best algorithm for performing a job is less optimal than a different algorithm, even if no human-driven changes have occurred in the system itself. Selection of a less-optimal algorithm may have serious detrimental consequences, since the cost of computational resources for these types of jobs is very high, and processing times or algorithmic correctness may also deteriorate meaningfully when data size is massive.

[0002] Prior approaches involved manually written rules-based (i.e., heuristic) approaches to determine algorithmic tradeoffs, but these can be very inaccurate because job behaviors evolve as relevant inputs change in unanticipated ways. For example, the relationship between inputs may change, such as the overlap rate for a job performing a big data join. The number of concurrent jobs on the same infrastructure may change. And other hard-to-predict, interrelated factors may evolve over time. What might have been perfectly good code last week might no longer be the optimal way to determine job algorithms.

[0003] Even worse, developers are very often unable to determine, even through involved analysis, rules that can effectively approximate the underlying phenomena that dictate optimal big data algorithm selection. Thus it becomes infeasible to attempt to make a potential tradeoff in question, because the penalties of getting it wrong and the frequency of getting it wrong with rules-based approaches would make it infeasible. So the result is that those persons running these jobs become systematically conservative in their tradeoff decisions, or avoid them altogether. By always opting for a particular choice and avoiding optimizations, the potential for lost savings and wasted compute resources is very high, as well as lost time as viable algorithmic speedups are not attempted.SUMMARY OF THE INVENTION

[0004] To address the problem of selecting an optimal algorithm for big data jobs, the invention is directed to a system and method using online machine learning that models user-Attorney Docket No. RAMP-00306-WO configured behaviors and phenomena from instrumented systems. As a result, subsequent jobs can be automatically and dynamically improved in a variety of ways, not necessarily limited to just infrastructure cost reduction, job time speedup, and accuracy.

[0005] By using online machine learning to predict when a job will have a specific behavior, as opposed to using heuristics, systems perform better because the optimal algorithm for the job at hand will be used more often. Furthermore, more aggressive optimizations can be made, because there is a higher confidence that the job will be a good match for the chosen algorithm. This improves over the approach of using heuristics, where it is harder to guarantee that a job is configured appropriately.

[0006] It will be seen then that the use of online machine learning for job planning in this manner leads to improved outcomes in terms of computational speed, the use of fewer computational resources, and lower cost.

[0007] The invention is especially useful in cases where there are no straightforward or deterministic methods to determine the best algorithm prior to job runtime. This is especially important in cases where the jobs in question are very large in scale, as the costs, resource usage, and business risks of making incorrect tradeoff decisions are much more significant in such circumstances. Often, systems with complex interactions between multiple inputs or constituent systems are difficult to predict deterministically, so this online machine-learning planner approach can provide better system behavior than static logic.

[0008] The invention can more accurately decide which pre-conceived big data algorithmic strategies will best serve upcoming jobs when it is otherwise infeasible to make this decision in a preprogrammed fashion. By more reliably matching each new job to the correct algorithms, the invention can directly result in better average job processing time, correctness, and very significant cloud resource usage reductions. This leads to financial, energy, and resource economies.

[0009] These and other features, objects and advantages of the present invention will become better understood from a consideration of the following detailed description of the preferred embodiments and appended claims in conjunction with the drawings as described following:BRIEF DESCRIPTION OF DRAWINGS

[0010] Fig. 1 is a diagram showing a data flow for a big data job according to an embodiment of the present invention.Attorney Docket No. RAMP-00306-WO

[0011] Fig. 2 is a diagram showing an overall architecture of a job planner used in conjunction with a big data job according to an embodiment of the present invention.

[0012] Fig. 3 is a schematic for a computing component of a computing cluster for implementing an embodiment of the present invention.DETAILED DESCRIPTION OF THE INVENTION

[0013] Before the present invention is described in further detail, it should be understood that the invention is not limited to the particular embodiments described, and that the terms used in describing the particular embodiments are for the purpose of describing those particular embodiments only, and are not intended to be limiting, since the scope of the present invention will be limited only by the claims.

[0014] A general data flow in a big data job is shown in Fig. 1, in connection with a job planner used in connection with the job flow according to an embodiment of the present invention as illustrated in Fig. 2. The job planner is a closed-loop automated decision maker that enhances big data systems by continuously modeling their relevant behaviors to automatically inform algorithmic decisions for subsequent jobs.

[0015] The job planner models relevant, user-configurable, big data system behavior and performance, both domain-specific as well as generic, to enable better algorithmic decisionmaking for big data jobs. This vastly improves many job characteristics, such as correctness, failure rate, cost, and speed. The planner achieves this by continually capturing and training on job metadata that is available before job runtime, as well as ground-truth job behavior metadata after jobs are attempted. The user of the invention configures what data is usable for predictions, which must be available before the job begins, and which behaviors should be predicted in subsequent jobs, which are continually modeled from past job behavior. The invention thus improves automatically over time as more metadata is gathered for machinelearning training.

[0016] There are a number of example cases that can be given where the job planner can inform job strategy. First, the planner can determine if a job, given learned domain specific models and specific new inputs, will take excessively long such that it needs to be distributed and run on many computers in a distributed computing cluster. Second, the planner can determine if a job, given learned domain specific models and specific new inputs, will be best served through one of several domain-specific algorithms. For example, it may choose among several potential traversal algorithms for graph traversal. Third, the planner can determine if aAttorney Docket No. RAMP-00306-WO job, given learned domain specific models and specific new inputs, will likely soon share input data with another yet-to-be-received job, such that related jobs can be more efficiently run in a batch mode. Fourth, the planner can determine if a job, given learned domain specific models and specific new inputs, is likely eligible for probabilistic algorithmic shortcuts. Such shortcuts might include using Bloom filters as opposed to a full big data join.

[0017] In some domains, probabilistic data structures, which come with a margin of error, might be essential most of the time, due to resource constraints; however, in some especially high-risk or idiosyncratic cases, as determinable by the planner, backup approaches might be taken. The inverse can also be true, where a deterministic algorithm might be chosen in most cases, but the planner can detect cases where algorithmic speed-ups and shortcuts are likely to be safe or especially important or both, as in cases where cost would otherwise be prohibitive.

[0018] There are many conceivable job behavior predictions that can directly inform algorithmic decisions for large scale jobs, such as whether to use distributed computing, how to consolidate requests into shared jobs and infrastructure, as well as even algorithmic correctness if it can better inform when and where to use certain strategies. All of these decisions can have material impacts on relevant system behavior, such as infrastructure cost, processing time, stability, and correctness.

[0019] A number of components of this embodiment of the invention work in concert to achieve automatic, enhanced algorithmic decision making. The automatic predictive metadata collection utility collects and aggregates relevant metadata. This metadata is knowable before a job is executed, on which machine-learning models can base predictions for optimal resource planning.

[0020] The automatic job behavior and performance metadata collection utility collects the realized performance and system behavior that corresponds to each set of predictive metadata, on which a model can be built, and which are knowable before jobs are run.

[0021] The online machine learning model training system automatically incorporates new real- world big data job examples to ensure that its predictions do not diverge from new trends which might affect real-world system performance. For example, changes in patterns in input data or availability of specific cloud resources, like multi-tenant preemptible or spot instances, whose stability differs through time, as well as seasonal changes in inputs or customer demand.

[0022] The machine learning prediction server provides user-configurable behavior predictions for jobs with specific predictions to inform the scheduling of subsequent jobs, optimizingAttorney Docket No. RAMP-00306-WO resource usage and ensuring reliability and good business results. This is the primary external interface of this embodiment of the invention, as client applications connect to this server to decide how to configure their applications based on predictions for the invention.

[0023] In order to most effectively use the embodiment as described herein, there must be sufficient training examples for the modeling components to derive useful models of big data performance, which requires frequent jobs to be run and modeled. In one real-world example, a team may aggregate metadata from over 100,000 such jobs a day, for example, in order to provide a powerful set of constant training data for the machine-learning system.

[0024] Still with reference to Figs. 1 and 2, a method for using the planner according to this embodiment of the invention may be described. Sequences of new inputs for processing are received for a series of jobs, denoted as inputs 10a, 10b... lOn. These inputs include metadata pertinent to each job. At pre-execution algorithm selection 13, an application programming interface (API) call is made through job algorithm planner API 17 in order to retrieve the particular algorithm that will be used for this particular big data job.

[0025] Following the API 17 call, operation is passed to the main system coordination server 20.Main system coordination server 20 acts as a controller for the whole process of determining the appropriate algorithm for a particular job. It sends job input metadata received through API 17 on to job algorithm prediction server 19. Job algorithm prediction server 19 in turn queries the database containing previously trained models of system behavior 26.

[0026] Next, trained models of system behavior 26 returns job behavior prediction information to job algorithm prediction server 19. This information is, in turn, passed on from job algorithm prediction server 19 to main system coordination server 20. The job behavior prediction information is passed back through API 17 to pre-execution algorithm selection 13, and thus the big data job now has the recommended algorithm needed in order to execute the job. The big data job is then executed at job execution 14.

[0027] Once the job is complete, job behavior metadata persistence 15 creates job behavior metadata from the actual results of running the job. This includes information necessary in order to determine for subsequent jobs which algorithm may be most appropriate. This information is passed by a call at API 17 to main system coordination server 20. In this case, main system coordination server 20 passes the job input and job behavior metadata on to job metadata coordinator and model training scheduler 18. The purpose of this component is to use the results of each big data job to train the system in a continuous process, therebyAttorney Docket No. RAMP-00306-WO iteratively improving the system as each new big data job is implemented, and allowing the system to adapt automatically over time as the nature of jobs and resources within the overall system fluctuates. Job metadata is stored at job metadata storage 22 for further use.

[0028] Job metadata coordinator and model training scheduler 18 calls on online machine learning model training component 24 in order to perform the actual training of the models. This process uses job metadata storage 22 as input. Once online machine learning model 24 completes its work, the resulting trained model is written out to trained models of system behavior 26. In this way, trained models of system behavior 26 are continuously updated as new big data jobs are run.

[0029] It may be noted that the processing that takes place following receipt of job behavior metadata from job behavior metadata persistence 15 can take place asynchronously with the actual performance of the big data job execution. That is because the queries from job algorithm prediction server 19 to trained models of system behavior 26 are not necessarily waiting on the development of a new model to be run. The recommendation is made from those models developed from earlier big data jobs that were run. The whole training process can be run on any schedule desired, giving greater flexibility to the system. On the other hand, the processing that takes place after pre-execution algorithm selection 13 sends job input metadata through API 17 to main system coordination server 20 must take place synchronously, since the big data job is waiting upon completion of that processing before an algorithm is recommended so that actual execution can take place at job execution 14.

[0030] As a prerequisite for deployment of this embodiment of the invention, hardware must be sufficient to train a machine-learning model and to serve predictions at the sufficient scale to inform subsequent job scheduling and to continually improve the model.

[0031] In alternative embodiments, the invention could be reconfigured to optimize for other desired outcomes related to large scale compute job configuration, rather than performance prediction and optimization.

[0032] The methods described herein may in various embodiments be implemented by any combination of hardware and software. For example, in one embodiment, the methods may be implemented by a computer system (e.g., a computer system as in Fig. 3) or a collection of computer systems, each of which includes one or more hardware processors executing program instructions stored on a computer-readable physical storage medium coupled to the hardware processors. These computer systems may be part of a cloud computing networkedAttorney Docket No. RAMP-00306-WO environment. The program instructions may implement the functionality described herein (e.g., the functionality of various hardware servers and other components that implement the network-based cloud and non-cloud computing resources described herein). The various methods as illustrated in the figures and described herein represent example implementations. The order of any method may be changed, and various elements may be added, modified, or omitted.

[0033] Fig. 3 is a block diagram illustrating an example computer hardware system, according to various embodiments. Computer system 140 may implement a hardware portion of a cloud computing system as forming parts of the various implementations of the present invention. Computer system 140 may be any of various types of hardware devices, including, but not limited to, a commodity server, personal computer system, desktop computer, laptop or notebook computer, mainframe computer system, handheld computer, workstation, network computer, a consumer device, application server, physical storage device, telephone, mobile telephone, or in general any type of computing node, compute node, compute device, and / or hardware computing device.

[0034] Computer system 140 includes one or more hardware processors 140a, 141b... Mln (any of which may include multiple processing cores, which may be single or multi-threaded) coupled to a physical system memory 142 via an input / output (I / O) interface 144. Computer system 140 further may include a network interface 146 coupled to I / O interface 144. In various embodiments, computer system 140 may be a single processor system including one hardware processor 141a, or a multiprocessor system including multiple hardware processors 141a, 141b... Mln as illustrated in Fig. 3.

[0035] Processors 141a, etc. may be any suitable processors capable of executing computing instructions. For example, in various embodiments, processors 141a, etc. may be general- purpose or embedded processors implementing any of a variety of instruction set architectures. In multiprocessor systems, each of processors 141a, etc. may commonly, but not necessarily, implement the same instruction set. The computer system 140 also includes one or more hardware network communication devices (e.g., network interface 146) for communicating with other systems and / or components over a communications network, such as a local area network, wide area network, or the Internet. For example, a client application executing on system 140 may use network interface 146 to communicate with a server application executing on a single hardware server or on a cluster of hardware servers that implement one or more ofAttorney Docket No. RAMP-00306-WO the components of the systems described herein in a cloud computing environment as implemented in various sub-systems. In another example, an instance of a server application executing on computer system 140 may use network interface 146 to communicate with other instances of an application that may be implemented on other computer systems.

[0036] In the illustrated embodiment, computer system 140 also includes one or more physical persistent storage devices 148 and / or one or more I / O devices 150. In various embodiments, persistent storage devices 148 may correspond to disk drives, tape drives, solid-state memory or drives, other mass storage devices, or any other persistent storage devices. Computer system 140 (or a distributed application or operating system operating thereon) may store instructions and / or data in persistent storage devices 148, as desired, and may retrieve the stored instructions and / or data as needed. For example, in some embodiments, computer system 140 may implement one or more nodes of a control plane or control system, and persistent storage 148 may include the solid-state drives (SSDs) attached to that server node. Multiple computer systems 140 may share the same persistent storage devices 148 or may share a pool of persistent storage devices, with the devices in the pool representing the same or different storage technologies, including such technologies as described above.

[0037] Computer system 140 includes one or more physical system memories 142 that may store code / instructions 143 and data 145 accessible by processor(s) 141a, etc. The system memories 142 may include multiple levels of memory and memory caches in a system designed to swap information in memories based on access speed, for example. The interleaving and swapping may extend to persistent storage devices 148 in a virtual memory implementation, where memory space is mapped onto the persistent storage devices 148. The technologies used to implement the system memories 142 may include, by way of example, static randomaccess memory (RAM), dynamic RAM, read-only memory (ROM), non-volatile memory, solid- state memory, or flash-type memory.

[0038] As with persistent storage devices 148, multiple computer systems 140 may share the same system memory systems 142 or may share a pool of system memories 142. System memory or memory systems 142 may contain program instructions 143 that are executable by processor(s) 141a, etc. to implement the routines described herein.

[0039] In various embodiments, program instructions 143 may be encoded in binary, Assembly language, any interpreted language such as Java, compiled languages such as C / C++, or in any combination thereof; the particular languages given here are only examples. In someAttorney Docket No. RAMP-00306-WO embodiments, program instructions 143 may implement multiple separate clients, server nodes, and / or other components.

[0040] In some implementations, program instructions 143 may include instructions executable to implement an operating system (not shown), which may be any of various operating systems, such as UNIX, LINUX, Solaris™, MacOS™, or Microsoft Windows™. Any or all of program instructions 143 may be provided as a computer program product, or software, that may include a non-transitory computer-readable storage medium having stored thereon instructions, which may be used to program a computer system (or other electronic devices) to perform a process according to various implementations. A non-transitory computer-readable storage medium may include any mechanism for storing information in a form (e.g., software or processing application) readable by a machine (e.g., a physical computer).

[0041] Generally speaking, a non-transitory computer-accessible medium may include computer-readable storage media or memory media such as magnetic or optical media, e.g., disk or DVD / CD-ROM, coupled to or in communication with computer system 140 via I / O interface 144. A non-transitory computer-readable storage medium may also include any volatile or non-volatile media such as RAM or ROM that may be included in some embodiments of computer system 140 as system memory 142 or another type of memory. In other implementations, program instructions may be communicated using optical, acoustical or other form of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.) conveyed via a communication medium such as a network and / or a wired or wireless link, such as may be implemented via network interface 606. Network interface 146 may be used to interface with other devices 142, which may include other computer systems or any type of external electronic device.

[0042] In some embodiments, system memory 142 may include data store 145, as described herein. In general, system memory 142 and persistent storage 148 may be accessible on other devices 142 through a network and may store data blocks, replicas of data blocks, metadata associated with data blocks, and / or their state, database configuration information, and / or any other information usable in implementing the routines described herein.

[0043] In one embodiment, I / O interface 144 may coordinate I / O traffic between processors141a, etc., system memory 142, and any peripheral devices in the system, including through network interface 146 or other peripheral interfaces. In some embodiments, I / O interface 144 may perform any necessary protocol, timing or other data transformations to convert dataAttorney Docket No. RAMP-00306-WO signals from one component (e.g., system memory 142) into a format suitable for use by another component (e.g., processors 141a, etc.).

[0044] In some embodiments, I / O interface 144 may include support for devices attached through various types of peripheral buses, such as a variant of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard, as examples. Also, in some embodiments, some or all of the functionality of I / O interface 144, such as an interface to system memory 142, may be incorporated directly into processor(s) 141a, etc.

[0045] Network interface 146 may allow data to be exchanged between computer system 140 and other devices attached to a network, such as other computer systems (which may implement one or more storage system server nodes, primary nodes, read-only node nodes, and / or clients of the database systems described herein), for example. In addition, I / O interface 144 may allow communication between computer system 140 and various I / O devices 150 and / or remote storage 148. Input / output devices 150 may, in some embodiments, include one or more display terminals, keyboards, keypads, touchpads, scanning devices, voice or optical recognition devices, or any other devices suitable for entering or retrieving data by one or more computer systems 140. These may connect directly to a particular computer system 140 or generally connect to multiple computer systems 140 in a cloud computing environment, grid computing environment, or other system involving multiple computer systems 140.

[0046] Multiple input / output devices 150 may be present in communication with computer system 140 or may be distributed on various nodes of a distributed system that includes computer system 140. In some embodiments, similar input / output devices may be separate from computer system 140 and may interact with one or more nodes of a distributed system that includes computer system 140 through a wired or wireless connection, such as over network interface 146. Network interface 146 may commonly support one or more wireless networking protocols (e.g., Wi-Fi / I EEE 802.11, or another wireless networking standard).

[0047] Network interface 146 may support communication via any suitable wired or wireless general data networks, such as other types of Ethernet networks, for example. Additionally, network interface 146 may support communication via telecommunications / telephony networks such as analog voice networks or digital fiber communications networks, via storage area networks such as Fibre Channel SANs, or via any other suitable type of network and / or protocol. In various embodiments, computer system 140 may include more, fewer, or differentAttorney Docket No. RAMP-00306-WO components than those illustrated in Fig. 2 (e.g., displays, video cards, audio cards, peripheral devices, or an Ethernet interface).

[0048] Any of the distributed system embodiments described herein, or any of their components, may be implemented as one or more network-based services in the cloud computing environment. For example, a read-write node and / or read-only nodes within the database tier of a hardware database system may present database services and / or other types of physical data storage services that employ the distributed storage systems described herein to clients as network-based services.

[0049] In some embodiments, a network-based service may be implemented by a software and / or hardware system designed to support interoperable machine-to-machine interaction over a network. A web service may have an interface described in a machine-processable format. Other systems may interact with the network-based service in a manner prescribed by the description of the network-based service's interface. For example, the network-based service may define various operations that other systems may invoke, and may define a particular application programming interface (API) to which other systems may be expected to conform when requesting the various operations.

[0050] In various embodiments, a network-based service may be requested or invoked through the use of a message that includes parameters and / or data associated with the network-based services request. Such a message may be formatted according to a particular markup language such as Extensible Markup Language (XML), and / or may be encapsulated using a protocol. To perform a network-based services request, a network-based services client may assemble a message including the request and convey the message to an addressable endpoint (e.g., a Uniform Resource Locator (URL)) corresponding to the web service, using an Internet-based application layer transfer protocol such as Hypertext Transfer Protocol (HTTP).

[0051] Unless otherwise stated, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0052] Although any methods and materials similar or equivalent to those described herein can also be used in the practice or testing of the present invention, a limited number of the exemplary methods and materials are described herein. It will be apparent to those skilled in the art that many more modifications are possible without departing from the inventive concepts herein.Attorney Docket No. RAMP-00306-WO

[0053] All terms used herein should be interpreted in the broadest possible manner consistent with the context.

[0054] When a grouping is used herein, all individual members of the group and all combinations and sub-combinations possible of the group are intended to be individually included.

[0055] When a range is stated herein, the range is intended to include all sub-ranges within the range, as well as all individual points within the range.

[0056] When "about," "approximately," or like terms are used herein, they are intended to include amounts, measurements, or the like that do not depart significantly from the expressly stated amount, measurement, or the like, such that the stated purpose of the apparatus or process is not lost.

[0057] All references cited herein are hereby incorporated by reference to the extent that there is no inconsistency with the disclosure of this specification.

[0058] The present invention has been described with reference to certain preferred and alternative embodiments that are intended to be exemplary only and not limiting to the full scope of the present invention, as set forth in the appended claims.

Claims

Attorney Docket No. RAMP-00306-WOClaims1. A method for optimizing big data job execution, comprising: receiving, at an input preparation server, a sequence of inputs for a big data job; sending from a pre-execution algorithm selection server, through an application programming interface (API), a request to determine an algorithm for executing the big data job; obtaining, from the API request, input metadata associated with the big data job; querying, by a job algorithm prediction server, a model database using the input metadata to obtain a job behavior prediction; selecting, based on the job behavior prediction, an algorithm for executing the big data job; providing the selected algorithm through the API for execution of the big data job; and storing job behavior metadata from execution of the big data job for subsequent model training.

2. The method of claim 1, further comprising the steps of: training, by a machine learning model training system, at least one machine learning model using the job behavior metadata; and storing the at least one trained machine learning model in the model database.Attorney Docket No. RAMP-00306-WO3. The method of claim 2, wherein the job behavior metadata comprises at least one of: job processing time; computational resource usage; job failure rate; or job accuracy metrics.

4. The method of claim 2, wherein the step of selecting the algorithm comprises determining whether to distribute the big data job execution across multiple computers; execute the job using domainspecific algorithms; batch the job with other related jobs; or apply probabilistic algorithmic shortcuts.

5. The method of claim 2, wherein training the at least one machine learning model occurs asynchronously with respect to the big data job execution.

6. The method of claim 2, further comprising: determining a number of concurrent jobs executing on shared infrastructure; and selecting the selected algorithm based at least in part on the number of concurrent jobs.

7. The method of claim 2, further comprising: evaluating system infrastructure health metrics as part of the input metadata; and selecting the selected algorithm based at least in part on the infrastructure health metrics.

8. The method of claim 2, wherein selecting the selected algorithm comprises determining whether to use a probabilistic data structure based on resource constraints and risk factors associated with the big data job.

9. The method of claim 8, wherein the probabilistic data structure comprises a Bloom filter used in place of a full join operation.

10. The method of claim 2, further comprising:Attorney Docket No. RAMP-00306-WO determining that the big data job likely shares input data with another pending job; and scheduling the big data job and the pending job for batch execution.

11. A system for optimizing big data job execution, comprising: one or more processors; memory coupled to the one or more processors and storing instructions that, when executed, cause the system to: implement a job algorithm planner API configured to receive job metadata and algorithm requests; implement a machine learning model training system configured to train at least one machine learning model using the job metadata; maintain a model database storing the trained machine learning models; implement a job algorithm prediction server configured to receive, through the API, requests to determine algorithms for executing big data jobs, obtain input metadata for the big data jobs, query the model database using the input metadata to obtain job behavior predictions, and select algorithms for executing the big data jobs based on the predictions; and implement a coordination server configured to provide the selected algorithms through the API, receive job behavior metadata from execution of the big data jobs, and store the job behavior metadata for subsequent model training.Attorney Docket No. RAMP-00306-WO12. The system of claim 11, wherein the job algorithm prediction server is further configured to determine whether to: distribute the big data job execution across multiple computers in a computing cluster; select domain-specific algorithms for the big data job execution; consolidate multiple jobs for batch execution; or apply probabilistic algorithmic shortcuts.

13. The system of claim 11, wherein the machine learning model training system is configured to train the models asynchronously with respect to the big data job execution.

14. The system of claim 11, wherein the metadata collection utility is configured to collect at least one of: job processing time metrics; computational resource usage metrics; job failure rate metrics; or job accuracy metrics.

15. The system of claim 11, further comprising a job metadata coordinator and model training scheduler configured to initiate new training runs for the machine learning models, and coordinate storage of new job metadata in the model database.

16. The system of claim 11, wherein the coordination server is further configured to monitor trends in job behavior across multiple executions, and trigger model retraining when the trends indicate changes in optimal algorithm selection.

17. The system of claim 11, wherein the machine learning model training system is configured to: maintain separate models for different types of big data operations; and select appropriate models to use based on characteristics of received job requests.

18. The system of claim 11, wherein the job algorithm prediction server is configured to: evaluate multiple candidate algorithms for executing the big data jobs; predict execution characteristics for eachAttorney Docket No. RAMP-00306-WO candidate algorithm; and select optimal algorithms based on predicted execution characteristics and job requirements.

19. The system of claim 11, wherein the job algorithm prediction server is further configured to determine whether any one of the big data jobs share input data with another pending job, and schedule such one big data job and the pending job for batch execution.

20. The system of claim 11, wherein job algorithm prediction server is further configured to determine whether to use a probabilistic data structure based on resource constraints and risk factors associated with the big data jobs.

Images