Agentic workflow system and method for artificial intelligence model aligned with domain-specific principles

Abstract

An agentic workflow system and method generate question and answer pairs and prompts that may be used to aligns generative artificial intelligence (a large language model (LLM) or a large multimodal model (LMM)) with the principles of a specific domain so that the generative artificial intelligence is better able to respond to a user query in the specific domain. The system and method may also generate aligning processes that may be used to post-train an already trained generative artificial intelligence system or fine tune the training of the generative artificial intelligence system to align that generative artificial intelligence system with the principles of the specific domain. The system and method may be used to align the generative artificial intelligence system to a plurality of different domains.

Description

AGENTIC WORKFLOW SYSTEM AND METHOD FOR ARTIFICIAL INTELLIGENCE MODEL ALIGNED WITH DOMAIN-SPECIFIC PRINCIPLESAPPENDICES

[0001] Appendix A (3 pages) is a document of the Global Alliance for Responsible Media (GARM) domain and its principles. Appendix B (4 pages) is a document of the civility domain and it principles. Appendix C (15 pages) is an example document tree for an example document that is an airline customer service plan. These appendices are part of the specification.PRIORITY CLAIMS / RELATED APPLICATIONS

[0002] This PCT patent application claims priority under the Paris Convention and 35 USC §§ 119 and 365 to US Patent Application Serial No. 19 / 085,323 filed March 20, 2025, US Patent Application Serial No. 19 / 086,631, filed March 21, 2025, US Patent Application Serial No. 19 / 088,106 filed March 24, 2025 (now US 12,499,387 issued on December 16, 2025), and US Patent Application Serial No. 19 / 089,576 filed March 25, 2025, all of which are continuations of and claim priority to claims priority under 35 USC 120 to U.S. Patent Application Serial No. 19 / 008,444 filed on January 2, 2025 (now US 12,293,272 issued on May 6, 2025), the entirety of all of which are incorporated by reference and form part of the specification.FIELD

[0003] The disclosure relates generally to artificial intelligence models (including large multimodal models (LMMs) and large language models (LLMs)) and in particular to one or more solutions to address problems and issues with current artificial intelligence models to provide systems and method for responsible artificial intelligence models.BACKGROUND

[0004] Artificial intelligence (Al) and machine learning are well known and have been used is various contexts for many years. Generative artificial intelligence (also generative Al or GenAI) is a new form of Al that is capable of generating text, images, or other media, using generative models. Generative Al models learn patterns and structure that are present in their input training data and then generate new data with similar characteristics and properties. Importantly, Generative Al models do not merely generate copies of their training data but learn to generalize and excel on tasks that require complex reasoning and commonsense steps that may have not been present in their training dataset. There are different types of GenAI that may include large language models (LLMs) that are text content-based or large multimodal models (LMM) that use all types of content and modalities (hence the multimodal name).

[0005] The introduction of a transformer-based architecture in 2018 enabled significant breakthroughs in Generative Al. Transformer-based models have unprecedented capabilities of modeling sequential data, due to their attention-based mechanism. The capability of Transformer models to scale have also added to their increased and ongoing popularity. There are currently several notable commercially available systems such as large language model chatbots, like OpenAI® ChatGPT, Microsoft® Bing Chat, Google® Bard, and Meta® LLaMA, and text-to-image artificial intelligence art systems such as Stable Diffusion, Midjoumey, and DALL-E. Generative Al has uses across a wide range of industries, including art, writing, software development, product design, healthcare, finance, gaming, marketing, and fashion.

[0006] One type of generative Al is powered by a large language model (LLM) that is a type of language model notable for its ability to achieve general -purpose language understanding and generation. Each LLM acquires these abilities by using massive amounts of data to learn billions of parameters during training and consuming large computational resources during their training and operation. LLMs are artificial neural networks (mainly Transformers) and are (pre-) trained using self-supervised learning and semi-supervised learning. Language models, including LLMs, work by taking an input text and repeatedly predicting the next token or word. Notable current examples of LLM systems includeOpenAI's GPT models (e g., GPT-3.5 and GPT-4, used in ChatGPT), Google's PaLM (used in Bard), and Meta's LLaMa, as well as BLOOM, Ernie 3.0 Titan, and Anthropic's Claude 2.

[0007] While LLM (and more generally, Generative Al) systems are incredibly powerful and useful in their ability to generate human-like content and assist with a wide range of tasks, an artificial intelligence system also raise a number of important ethical, moral, social, and technical considerations. For example, it is widely known that LLMs often generate content that may be factually incorrect, toxic, harmful, and deceiving. Some current LLMs use a known reinforcement Learning with Human Feedback (RLHF) technique to overcome some of the above issues, but RLHF alone, often does not fully resolve the issue. While foundation LLMs can perform remarkably well in a broader context, they lack the domain¬ specific knowledge to be helpful in most industrial or business applications. Generically pre¬ trained LLMs have known limitations when asked to perform tasks that may require adherence or compliance to principles of a specific domain. For example, domains such as banking, retailjournalism, legal, financial, medical, pharmaceutical, education, etc. have certain standards, intentions, rules or values that specify what constitutes acceptable model behavior. Generically pre-trained foundation LLMs may not have knowledge of the domainspecific organizational guidelines, standards, rules, intentions or values; these are collectively defined as domain principles. Hence, foundation LLMs will be challenged in well-defined tasks dictated by domain principles and there are several technical problems with safely applying them ubiquitously.No domain information

[0008] First, most foundation LLMs, are pre-trained on massive corpora that may be missing crucial and recent world knowledge and information. Typically, the training corpora are just snapshots of data collected at a specific moment in time; this limits LLMs utility, especially when the domain-specific context may not exist in the pre-training corpora.Information conflicting with domain-specific principles

[0009] Second, because LLM pre-training generally ignores domain-specific principles, it can lead to the LLM model having seen information in its training set that is in conflict with the domain-specific principles. In this case, even elaborate prompt engineeringtechniques, including few-shot, Chain-of-Thought, Tree-of-Thoughts etc. are not enough to guarantee that the LLM behaves within the scope of the domain. The conflict between what the model has seen in its training set and the domain-specific principles will lead to an averaging effect and will cause the model to produce a low-confidence signal. Hence, alignment is necessary to ensure that the model forgets the conflicting knowledge and enforce the model to leam the domain-specific signals.Tangentially related domain-information

[0010] In addition, whatever potential knowledge the LLM may have, it will have obtained it from the pre-training data, which may only be tangentially related to the principles in each specific domain.Unclear understanding of domain-principles

[0011] The foundation LLMs do not have a clear understanding of the principles, how they are defined and the specific criteria that indicate appropriate behavior.Prompt Engineering and Inference

[0012] LLMs generate human-like text through a process known as generative inference in which, given an input prompt, generative LLM inference generates text outputs, by iteratively predicting the next token in a sequence. Complicated prompts required for typical unaligned LLMs are time consuming and expensive. Importantly, complicated prompts increase computation time and latency, making the models unusable for specific applications. Alignment ensures that the model will concisely leam the domain-specific task / s and eliminates the need for complicated, lengthy and multistep prompts.

[0013] In addition to the LLMs discussed above, large multimodal models (LMMs) also have the same issues and limitations. LMMs exist because text alone (the LLM) is low-bandwidth, scarce, and not rich enough to achieve human-level commonsense and intelligence: humans don’t leam just from text. To solve this problem and expand capabilities, the LMMs focus on additional modalities (such as image, video, and audio inputs, as well as text inputs). These LMMs, like the LLMs, if generically pre-trained, are not aligned with domain specific principles.

[0014] The above limitations and failures of known LLMs / Generative Al and LMMs are caused by technical problems with how LLMs and LMMs work and generate responses as described above. It is thus desirable to provide a framework, including systems and methods to generate instructions to train and align an artificial intelligence model and / or take an existing pre-trained LLM or LMM and post-train or fine-tune the LLM or LMM to ensure that it is aligned with the domain-specific principles and that it behaves within the rules, guidelines and ethics of the domain.

[0015] Besides alignment during training or fine-tuning, LLMs or LMMs may need to be aligned during inference. This may be achieved by the LLM being asked to validate its own response. This process forces the LLM to “think” harder about its output, revisit the chain-of-thought it took to arrive to a specific response and validate it against the domain-specific principles. This process is called post-editing and it may be required to ensure that the LLM produces responses that are aligned with the domain-specific principles.

[0016] Thus, the disclosure is directed towards providing a technical solution towards providing a LLM or LMM model that may be post-trained or fine-tuned during training with domain-specific knowledge, thus having a better understanding of the domain and being able to operate within the domain context more accurately and safely.BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a system that incorporates an LLM or LMM aligned with domain specific principles to generate responses to user’s prompts;

[0018] Figure 2 illustrates more details of a training module 106B of the system in Figure 1 that can post-train or fine tune the training of an LLM or LMM to align with different one or more domain specific principles;

[0019] Figure 3 illustrates a method for fine-tuning the training of an LLM or LMM;

[0020] Figure 4 illustrates a method and data flow for post-training an LLM or LMM to be aligned with domain specific principles;

[0021] Figure 5A illustrates an agentic workflow system being used to train and align an LLM / LMM to domain specific principles;

[0022] Figure 5B illustrates more details of the agentic workflow system and process to train and align the LMM / LLM to domain specific principles;

[0023] Figure 6A illustrates an agentic workflow method for generating a synthetic dataset of instructions;

[0024] Figure 6B illustrates the evolution of a hierarchical document tree during the agentic workflow method;

[0025] Figures 7A and 7B illustrate further details of the method for QA pair generation; and

[0026] Figure 8 illustrates a method for aligning an LLM / LMM using the synthetic dataset of instructions generated by the teacher LLM / LMM.DETAILED DESCRIPTION OF ONE OR MORE EMBODIMENTS

[0027] The disclosure is particularly applicable to systems and methods for aligning large multimodal models (LMMs) or large language models (LLMs) (collectively “artificial intelligence models”) with domain-specific principles using a well-known transformer architecture. The aligning of the artificial intelligence models involves using domainspecific principle data and documents to generate a plurality of instructions that are then used to align the artificial intelligence models. As discussed below, an agentic workflow may be used to generate the plurality of instructions (such as by using the synthetic data generating process discussed below) and the artificial intelligence model may be aligned by pre-training on domain specific text corpora and / or fine-tuned using training data messages (See Figures 2-3). Alternatively, the artificial intelligence model may be aligned by post inference alignment processes (See Figure 4) that use the plurality of generated instructions (generated by the synthetic data generating process discussed below) to post-train the artificial intelligence model. Post-training collectively refers to additional pre-training and / or finetuning on an already pre-trained model.

[0028] The resultant aligned LMM or LMM or the aligned responses from the LMM or LLM for a post-trained LLM or LMM are a technical solution to the problems and limitations with known LLMs and LMMs and is able to perform tasks, such as content generation, classification, chat, summarization etc. that comply with the specific domain’sprinciples and context. In one embodiment, the LMM or LLM will be trained using training instructions / data points that are generated for the domain-specific principles. It is noted that the system and method disclosed below may be used to additionally pre-train or fine-tune training for either an LLM or an LMM and both are within the scope of the disclosure. Each of the domain specific aligned LLM and domain specific aligned LMM make inference and prompting less complicated, much more efficient and cost-effective. For purposes of this disclosure, artificial intelligence (Al) refers to a generative Al system that may be implemented by an LLM or an LMM.

[0029] The system and method described below may be used to align an LMM or LMM with one or more different domain specific principles. Furthermore, the system and method may be used to train a single LLM or LMM to be aligned with two or more domain principles at the same time. In some embodiments, an LLM or LMM may be aligned by the system to domain specific principles either through continued pre-training or fine tuning on synthetically generated training data messages (See Figure 2).

[0030] The domain as used in this disclosure is a set of data that may be as input to drive the instruction generation process as discussed below wherein the generated instructions align the LLM or LMM to the particular domain. For example, the domain may be rules or standards for a particular industry or vertical (with examples provided below), a set of forms, policies, etc., for an organization, a set of files / data of a person’s computer or more than one person’s computer and / or an application data set including a set of blog posts.

[0031] The system and method described below for producing a domain-specific instruction set may be a training data messages-set generating agent. In an embodiment in which a plurality of instructions are used to train the artificial intelligence model to align that artificial intelligence model (see Figure 2), the training data messages-set generating agent may be an instruction generation agent 106B1 as shown in Figure 2. In an embodiment in which a plurality of instructions are used to inform the alignment processes 410 that post inference align a response from the artificial intelligence model, the training data messages-set generating agent may be the similar to the instruction generation agent 106B1 (shown in Figure 2, but not in Figure 4) In is noted that the training data messages-set may be used for other purposes that those shown in Figures 2 and 3. The training data messages-setgeneration agent 106B1 is able to plan the task of generating the training messages, according to the information that it is given that may include the domain data. The agent 106B1 may have access to one or more external tools, APIs, such as a search API, a code interpreter or a calculator, etc., to augment its knowledge. The agent’s planning involves processing, decomposing and understanding the user-provided domain principles and executing the training messages generation task(s). To generate the training messages set to align an LLM or LMM (either by training or post inference), the agent 106B1 may perform: retrieval: with access to search APIs and other tools, the agent will retrieve relevant documents from the web or other data sources and will transform them to instructions; and / or self-reflection and self-critique. For example, the agent 106B1 will generate critiques to criticize its own output, in terms of factuality, overall quality, and alignment with the domain principles as shown in Figure 5B. The generated critiques will be used to decide whether the generated training message is sensible in relation to the task and the domain-specific principles and whether it should be retained or discarded as discussed below in more detail.

[0032] The domain may be a set of rules, standard(s) for an industry, or a vertical domain. A well-known Global Alliance for Responsible Media (GARM) standard (See Appendix A) or a civility standard (See Appendix B) may be examples of the domain.Examples of further details for each of these domains, a user prompt, the LLM response provided by the domain principle trained LLM and the LLM response for known LLM models are provided below. Note that for illustration purposes, the example of the LLM trained for domain specific principles is an LLM and the known LLMs for comparison are Llama2 and GPT-4 which are well-known and understood publicly available LLMs whose content is text. Note that the LLM aligned with domain specific principles may be aligned to any of one or more domains and those domains are not limited to the three examples provided below. Furthermore, the domains may include a set of rules set by a third party, like GARM, a standard used in an industry, like a content standard such as civility and / or an industry or a specific area of focus, like agriculture. In addition, when an LLM is aligned to a particular domain’s principles, that LLM is trained about different data than a different LLM that is aligned with a different domain. Another example of the vertical domain is agriculture, legal or medical. Furthermore, the domain may be policies and documents for an entity or may be an application (e.g. someone’s computer or phone, a company’s internaldata, files etc.), e.g. files / data on device (e.g. laptop, desktop computer, phone, tablet, attached storage etc.); files / data on cloud / shared storage (e.g. shared drive); and / or files / data on an enterprise / business systems. Like the other domains above, the application is what will drive the training messages generation process that will be used to teach / align any pre-trained LLM.

[0033] A first domain example is GARM, that is a certain ruleset aimed at reducing harmful content in text-based content by establishing shared definitions, developing standardized reporting, and enhancing transparency among stakeholders. GARM’s mission is to ensure that digital media platforms are places where brands can advertise safely, without their ads appearing next to harmful or inappropriate content.GARM Domain ExampleExample Prompt Aligned LLM Llama2 response GPT-4 response ResponseI mean, you don’t. I Category: obscenity Category: debated Category: obscenity mean, nigga. It’s not and profanity, sensitive social and profanity, bad. Can white including language, issue; risk: high including language, people get waves? gestures, and gestures, and People get waves?” explicitly gory, explicitly gory,graphic or repulsive graphic or repulsive content intended to content intended to shock and disgust; shock and disgust; risk: high risk: floor (also predicted “high” in Category: hateanother instance) speech & acts ofaggression; risk: Category: hate high speech & acts of aggression; risk: floor

[0034] As shown in the above example, while GPT-4’s response is similar to the aligned LLM, its predicted risk is lower (floor vs. high) Llama2’s response missed both the categories identified by both the aligned LLM and GPT-4 which highlights the discrepancies that may exist in known LLMs in terms of alignment to domain specific principles. Further details of the GARM principles are in Appendix A that is incorporated herein by reference.

[0035] A second domain example is civility that is defined in terms of personal attacks and their severity. Specifically, any kind of insulting language toward an individual or group is considered an attack, including any of the following: Name-calling; Derogatory terms; Paraphrasing / quoting insults; Insinuating something insulting; Mocking through imitation; Threats / threatening language; and Self-deprecation. An attack is severe when it is based on characteristics with legal protections against discrimination. For instance, insulting someone based purely on a protected characteristic is considered a severe attack and those protected characteristics may include Race / color; Religion; National origin; Age (40 and over); Sex; Gender identity (incl. respecting someone’s expressed I preferred gender identity); Sexual orientation; Pregnancy; Familial status; Disability status; Veteran status; and / or Genetic information. If the attack is not based on any of the above legally protected characteristics, it is considered general.Civility Domain ExampleExample Prompt Aligned LLM Llama2 GPT-4Response response response You give, then all of a sudden you General Attack Severe Attack No Attack give them like the blm, you getmillions. And what do you do? Youkind of just prove it to people. Andyou did it on the international stage.All right, we’ll give you reparations.Whoop, you stole ‘em from eachother. Yeah. It’s almost like the jokes.Just write them f***ing selves at thispoint. <laugh>. I mean, you can’t callthat anecdotal. That’s real life. That’sreal world. Uh, examples, if you will,which is funny to me. This guy lookslike a f***ing nerd. Another little racegrifter out there. Joy Behar comes out.I don’t give a f*** about that bit**.All right, last, most certainly, not leastthe f*** Joy Behar. Hope she dies.F*** c**t. I try to be moreprofessional on this show, but, oh,these people just bring it outta me.And by people I mean Joy Behar, notblacks.

[0036] Like the GARM domain example above, the above example illustrates the differences in responses between the novel LLM aligned with domain principles and the known LLMs that mis-identify the attack and its severity with Llama2 mis-identifying the attack as severe (when it is not, although it is certainly offensive) while GPT-4 does not find any attack at all. The novel LLM aligned with domain principles has properly identified a civility attack but identified it as a general attack consistent with the civility principles noted above. Further details of the civility principles are in Appendix B that is incorporated herein by reference.

[0037] A third domain example is the subject of agriculture.Agriculture Domain ExampleExample Aligned LLM Llama2 response GPT-4 responsePrompt ResponseWhat is the best The best time to N / A The best time to plant trees and times to plant plant shrubs in shrubs in Connecticut is during trees and shrubs shady areas is the early spring and fall in Connecticut? during late winter seasons. Planting in early spring or early spring in (April to May) allows the plants well-prepared to establish themselves before soil. the summer heat arrives. Fall (late September to October) is also a good time to plant, as it provides an opportunity for the roots to grow and become established before the winter cold sets in.

[0038] Like the two other domain examples, this example shows the disparity in the responses between the known LLMs and the novel LLM aligned to the domain specific principles. While examples of the GARM, civility and agriculture domains are provided for illustration purposes, the system and method may be used to align the LLM or LMM to other domains with domain principles in which the domains may include banking, retail, journalism, legal, financial, medical, pharmaceutical, education etc.

[0039] Another example domain are personalization based on information from data stored on a personal device.Personalization based on information from data stored on a personal device Domain Example Aligned Llama2 Response GPT-4 Response Description Prompt ResponsePersonalization Where Based on your There are many Choosing the based on should I go most recent great places to go perfect vacation information on vacation vacation on vacation this spot for the summer from data this travels (i.e. last summer. Some can depend on stored on a summer? 5 years), I popular choices several factors, personal suggest that include: such as your device, you continue interests, budget, downloaded to explore 1. Europe: Explore and how far you're from different Southeast iconic cities like willing to travel. apps, fdes etc. Asia. You Paris, London, Here are a few have already Rome, and suggestions across visited Japan, Barcelona. different categories: The aligned China, andLLM is fineSouth Korea 2. United States: 1. Beachtuned on an several times. Discover national Destinations: Maui, instructions set How about parks like Hawaii: Ideal for that is rooted switching it up Yellowstone or stunning beaches, on specific a bit and Yosemite, or visit snorkeling, and a user data, visiting iconic cities like relaxed island vibe. previous app Vietnam or New York, Los Amalfi Coast, Italy: usage, Thailand? Angeles, or San Offers picturesque bookings, and Another visit Francisco. cliffs, exquisite browsing to Japan could Italian cuisine, and history. be nice as 3. Caribbean vibrant seaside well, perhaps Islands: Enjoy villages. during the warm weather andspring, during beautiful beaches 2. Adventure and cherry in destinations like Nature: Banff blossom Jamaica, Aruba, or National Park, season (based the Dominican Canada: Perfect for on your love Republic. hiking, mountain of plants). biking, and wildlife 4. Hawaii: spotting in the Experience natural majestic Canadian beauty and unique Rockies. Costa culture on the Rica: A great Hawaiian Islands. destination for eco- tourism, with 5. Africa: opportunities for Encounter wildlife zip-lining, jungle on a safari in tours, and surfing.countries likeDomain Example Aligned Llama2 Response GPT-4 Response Description Prompt ResponseKenya, Tanzania, 3. Cultural or South Africa. Exploration: Kyoto,Japan: Explore 6. Asia: Immerse traditional tea yourself in rich houses, ancient cultures and temples, and the diverse landscapes historic Gion in countries like district. Barcelona, Thailand, India, or Spain: Known for Japan. its unique architecture, lively 7. Australia: markets, and Discover stunning vibrant street life. landscapes4. Relaxing Retreats: Santorini, Greece: Famous for its dramatic views, stunning sunsets, and peaceful environment.Maldives: Ideal for luxury overwater bungalows, snorkeling, and diving in crystalclearwaters.5. Urban Escapes: New York City, USA: Offers Broadway shows, museums like the Metropolitan Museum of Art, and diverse cuisine. Paris, France:Perfect for lovers of art, fashion, and gourmet food, with landmarks like the Louvre and Eiffel Tower.Domain Example Aligned Llama2 Response GPT-4 Response Description Prompt ResponseThink about what type of activities you enjoy, what cultural experiences you might want to have, and whether you're looking for relaxation or adventure. This will help narrow down the choices to find the perfect destination for yoursummer vacation!

[0040] Another example domain are enterprise internal documents.Enterprise Internal DocumentsDomain Example Aligned Response Llama2 Response GPT-4 Description Prompt Response Enterprise Rank the top According to our 1. Technical Skills: When internal qualities one company's core Knowledge of considering documents. would want to values and the programming the top see in a junior desired traits for a languages (e.g., Java, qualities in a The software junior software Python, JavaScript)- junior aligned engineer engineer outlined in Familiarity with software LLM has interview our HR documents, development engineer, it's been candidate. the top qualities we frameworks and important to trained on seek in a candidate, libraries- Proficiency prioritize an ranked in order of in version control traits that not instruction importance, are ones systems (e.g., Git) only facilitate set that has that would immediate been particularly benefit a 2. Problem-solving contributions generated small, lean start-up: skills: Ability to but also longby analyze complex term growth processing Ownership and problems- Effective and internal Accountability: use of logical adaptation in company Takes responsibility reasoning- a team documents for their work, Comfortability setting. that specify delivers on working through Here’s a core commitments, and algorithmic solutions ranking of company strives for thesevalues and excellence. qualities,Domain Example Aligned Response Llama2 Response GPT-4 Description Prompt Response corporate 3. Communication starting with culture. Grit and Resilience: skills: Clear and the most Perseveres through concise verbal crucial: challenges, learns expression- Activefrom setbacks, and listening- Effective Problem- continuously collaboration within Solving improves. a team Skills: This is foundational Passion for 4. Adaptability: because Technology: Flexible attitude software Demonstrates towards new engineering is genuine enthusiasm technologies and primarily for software methodologies- about solving development and Ability to learn problems eagerness to learn quickly in a fast- using and grow. paced environment- technology. A Adapting to change candidate as needed who can think logically and 5. Time creatively to Management: solve Prioritizing tasks complex effectively- Working problems will under deadlines- be highly Managing workload valued. to meet projectmilestones Strong Foundation in Programming: Proficiency in key programming languages and principles underpins all coding tasks. A solid understanding of algorithms, data structures, and basicsoftwareDomain Example Aligned Response Llama2 Response GPT-4 Description Prompt Response architecture is essential.Willingness to Learn: The tech field is constantly evolving, and a good junior software engineer must be eager to continuously update their skills and adapt to newtechnologies.

[0041] Another example domain are blog posts.Blog PostsDomain Example Aligned Response Llama2 Response GPT-4 Description Prompt Response Blogpost Write a blog Introducing Introducing N / A announcing post for the SeekrFlow: SeekrFlow: Thethe release release of Redefining Trusted Simple Path toof a SeekrFlow. Al Development for Trustworthy Alsoftware All In today's fastcalled Seekrflow is evolving digitalSeekrFlow. the simple way April 18th, 2024 landscape, artificialto build, scale, intelligence (Al) has The and validate Today marks a become a cornerstonealigned trustworthy Al significant milestone of innovation.LLM’s Customizing in the evolution of However, the journeyresponse Al can be time artificial intelligence to build and scale Alfollows and cost with the launch of systems can often bestyle, tone, intensive. SeekrFlow, an fraught withgrammar SeekrFlow innovative platform complexity,and format helps designed to especially when itthat is developers transform the way comes to trainingrooted on train and developers build, models acrossthe prompt. validate scale, and validate various hardwareIt is also trustworthy Al trusted Al. By platforms and cloudmuch more using a addressing the ecosystems. But whatDomain Example Aligned Response Llama2 Response GPT-4 Description Prompt Response precise (i.e. workload-first unique challenges if there was amentions a approach — associated with Al simpler, moredate and eliminating the development, streamlined way tospecific complexity of SeekrFlow paves the handle this process? features) training models way for a future Enter SeekrFlow, aon different where creating and revolutionaryhardware maintaining platform designed to platforms and trustworthy Al is redefine howcloud straightforward and developers approach ecosystems, all efficient. Al, focusing onthrough a building, scaling, and simple A Workload-First validatinginterface. Approach to Trusted trustworthy AlAl solutions with ease.Simplifying the AlAt SeekrFlow, we Development Process understand that every At its core,Al project has its SeekrFlow is aboutown set of simplicity andrequirements and efficiency. Thechallenges. Our platform offers aplatform champions workload-firsta workload-first approach, whichstrategy, focusing on means that itthe specific needs of prioritizes theeach project right specific needs offrom the start. This your Al tasks before approach ensures anything else. Thisthat all tools and approach ensures that environments are developers can focus optimally aligned more on solvingwith the project problems and less ongoals, which the nuances of theenhances efficiency technology stack.and cuts down on Whether you're a development time startup looking toand costs. quickly deploy yourThe interoperability first Al model, or aof SeekrFlow across large enterprisevarious hardware and aiming to scalecloud platforms complex Aleliminates the operations,complexity often SeekrFlow providesfaced when training the tools necessary toDomain Example Aligned Response Llama2 Response GPT-4 Description Prompt Response Al models in make your journey different smoother.environments. By Cross-Platformproviding a flexible Compatibility Madeand cohesive Easyexperience, One of the biggest developers can challenges in Aloptimize their Al development issolutions without ensuring that modelsbeing tied to any perform consistently specific ecosystem. across differentBuilding and Scaling hardware and cloud Trustworthy Al environments.Trust and reliability SeekrFlow eliminates are at the core of this barrier bySeekrFlow. Our offering a seamlessplatform is equipped interface thatwith advanced integrates withvalidation tools that various platforms. rigorously test Al This not only reduces models for biases the time and costand errors, ensuring associated withthe output is not only model training buthigh-performing but also significantlyalso adheres to the lowers the entryhighest ethical barrier for Alstandards. This innovations.dedication to trusted Developers can nowAl helps in fostering train and deployresponsible models without implementation and having to worryconfidence in Al about the underlying solutions. hardware or cloud Scalability is critical specifics.in Al development, Trustworthiness inand SeekrFlow Alexcels in this regard. In an era where AlIt is built to grow ethics andwith your business, trustworthiness are effortlessly handling more important than increased demands ever, SeekrFlowwithout sacrificing places a strongperformance or emphasis on buildingsecurity. This reliable Al systems.Domain Example Aligned Response Llama2 Response GPT-4 Description Prompt Response scalability ensures The platform comesthat as your data equipped with toolsneeds evolve, your that help validate theAl solutions continue accuracy and fairness to deliver without of Al models,interruption. ensuring they adhere Simplifying Al to ethical standards Development with a and regulatoryUnified Interface requirements. This is SeekrFlow’s crucial forintuitive interface is organizations aiming designed to simplify to deploy Althe Al development solutions that are not process. By only effective but integrating all also sociallyessential tools and responsible.resources into a A User-Friendlysingle, accessible Interfaceplatform, developers SeekrFlow's user- can streamline their friendly interface workflows, focus on stands out, designed innovation, and with the developer's accelerate the experience in mind. deployment of It guides userstrusted Al solutions. through the process Embrace the Future of Al developmentof Trusted Al with from start to finish, SeekrFlow making it accessible SeekrFlow is more for both novicethan just a platform; programmers andit is a commitment to seasoned Al experts. making trusted Al With drag-and-drop development features and pre-built accessible to templates, SeekrFlow everyone. By makes it easier than breaking down the ever to bring Altraditional barriers projects to life.associated with Al Get Started withtraining and SeekrFlowseal ability, Ready to take your SeekrFlow is setting Al projects to thea new standard in the next level?field. SeekrFlow is here tohelp you build, scale,Domain Example Aligned Response Llama2 Response GPT-4 Description Prompt Response We invite and validatedevelopers, tech trustworthy Alcompanies, and Al applicationsenthusiasts to join us efficiently. Whetherin this exciting you're looking tojourney. Explore enhance yourwhat SeekrFlow has existing solutions orto offer and discover start a new projecthow it can transform from scratch,your Al projects. SeekrFlow providesVisit our website for the tools and supportmore information, you need to succeed. demonstrations, and Stay ahead of theto get started with curve and join thethe future of trusted growing communityAl development. of developers whoWelcome to the new trust SeekrFlow forera of Al with their Al development SeekrFlow, where needs. Experience the developing trusted, future of Al withscalable, and SeekrFlow, whereefficient Al solutions innovation meetsis easier than ever. simplicity.

[0042] Figure 1 is a system 100 that incorporates an LLM or LMM aligned with domain specific principles to generate responses to user’s prompts. As will be described below, the system 100 may be used to align an already trained (continued pre-training) LLM or LMM (See Figure 4) with one or more domain specific principles and / or fine tune the training of the LLM or LMM (See Figure 2) with one or more domain specific principles. The system may be used to train or post-train an LLM / LMM associated with the system 100 or may be used to train or post-train a third party LLM / LMM. The system may have one or more application programming interfaces (APIs) that may be used by third parties to submit an LLM / LMM for alignment, to receive the data to align their own LLM / LMM, to submit a query / receive a response from the system’s LLM / LMM aligned with particular domain specific principles and the like. In an embodiment in which the system has its own aligned LLM / LMM, the system may have a plurality of aligned LLMs / LMMs housed in the system 100 in which each LLM / LMM is aligned to a different domain and then may be selected by auser. Alternatively, the system 100 may have an LLM / LMM aligned to multiple different domain principles that may be used by the system. For the embodiment with post-training alignment of the LLM / LMM, the system may apply the alignments for the principles specific for each domain to generate the aligned responses using the existing / conventional LLM / LMM.

[0043] In an embodiment in which the aligned LLM / LMM is housed in a backend 106 of the system, the system 100 may receive inputs from one or more users over a computer network 104 that are received by a backend system 106 that, using a trained and aligned large language model (LLM) or large multimodal model (LMM) 106A, generates a response to the user that is delivered to the user via a user interface engine 106C of the backend system 106. Thus, each user may generate a query / prompt for the LLM / LMM 106A that may be generated on a computing device 102 of the user and then transmitted over the computer network 104 to the backend system 106. The trained and aligned LLM / / LMM 106A (that is aligned with domain specific principles and ethics and behaves with the rules, principles, guidelines and ethics of the particular domain) may generate a response based on the user prompt / query and return the response to the computing device 102 using the user interface engine 106C. Note that examples of the user prompts and responses from known LLMs and the trained and aligned LLM / LMM for the GARM domain, the civility domain and the agriculture domain are provided above.

[0044] In an embodiment in which a third party LLM / LMM is post-trained for alignment, the user may, using APIs, submit a prompt to the LLM / LMM as shown in Figure 4 and the results of the response to the prompt are adjusted to be aligned to the domain specific principles. The results, adjusted to the domain specific principles and thus aligned to those domain specific principles, are then returned to the user so that the user benefits from the LLM / LMM aligned to the domain specific principles as will be discussed in more detail below with reference to Figure 4.

[0045] The below disclosed system and method post-trains (see figure 4) or fine tunes the training (see Figures 2-3) of an LLM / LMM (system LLM / LMM or third party LLM / LMM) so that the LLM / LMM is aligned with one or more principles for specific domains and that is able to generate content that is based on the principles of the domain. For example, in thecase of the LLM / LMM aligned with journalistic principles, the LLM / LMM is able to detect and reason around the reliability, bias and violations of journalistic integrity found in the corpus of documents. In one implementation, the LLM / LMM may be trained with specific instructions that are crafted by trained experts in the particular domain to ensure the alignment with the principles of the specific domain. The aligned LLM / LMM, when aligned with journalistic principles, may use a massive training set of scored news articles with ratings on clickbait, exaggeration, subjectivity, source quality, dog whistle detection, political bias scored using the system’s algorithms and scoring technology. The aligned LLM / LMM also may use Reinforcement Learning with Human in the Loop (RLHL) techniques to guide the LLM / LMM towards producing content that is aligned with particular domain principles. The aligned LLM / LMM system and method may have prompt instructions that carefully explain inputs and outputs that align with the specific domain.

[0046] As shown in Figure 1, the system 100 allows a plurality of users to interact with the system 100 over the network 104. Each computing device 102 is a processing unit based device with inputs / outputs (one or more of a touch screen, keyboard, mouse, display, etc.,) and circuits for communicating over the network 104 with the backend 106. The user may interact using a browser application executed by the processor of the computing device 102 or using a mobile application or downloaded application that is also executed by the processor of the computing device 102. For example, each computing device 102 may be a laptop computer 102A, a tablet computer 102B, a personal computer 102C, , and / or a phone or smartphone device 102N, such as a Apple® iPhone® or a Google® Android® operating system based device.

[0047] The network 104 may use a communications protocol (TCP / IP) and a data transfer protocol (such as HTTP or HTTPS) or a yet to be developed communications and data transfer protocol to allow each computing device 102 and the backend system to connect to each other and communicate with each other. The network 104 may comprising one or more wired or wireless communication paths including Ethernet, a digital data network, a cellular data network, a cellular network, a WAN, a LAN, a backhaul network and the like.

[0048] The backend system 106 may be one or more computing resources including memory and processors that train and operate the trained LLM / LMM 106A or post-train anLLM / LMM and execute the user interface engine 106C. For example, the one or more computing resources may be cloud computing resources from Amazon AWS or Microsoft Azure, a server computer, an application computer and the like. In addition to the LLLM / LMM 106A and the user interface module 106C, the backend 196 may further comprise a training module 106B that post-trains or fine tunes the training on an LLM / LMM to align it with the principles of a specific domain (as discussed in more detail below with reference to Figure 2-4) and a content scoring system 106D that performs quality scoring on each piece of content that may be used by the system to align the LLM / LMM to domain specific principles as discussed below. In one embodiment, the trained LLM 106 A, the training module 106B, the user interface engine 106C and the content scoring system 106D may each be a plurality of lines of computer code / instructions executed by a processor of the backend computer system 106 that implement the processes of those systems wherein the processor is configured to perform those processes as discussed below. The backend system 106 may be coupled to one or more data stores 108A, ..., 108N that store theinstruct! ons / computer code for the LLM / LMM 106A and the user interface engine 106C, the corpus of data used to train the LLM / LMM 106A and various other data used by the system. In one implementation, one or more of the data stores may be known vector databases 108B that store embeddings associated with the LLM as described below.

[0049] In one implementation, the trained LLM / LMM 106A aligned with domain specific principles may be an LLM / LMM such as a Language Representation Model, such as GPT (Generative Pre-trained Transformer) models, BERT (Bidirectional Encoder Representations from Transformers), and RoBERTa, a Zero-shot Model, such as GPT-3, a Multimodal Model, such as OpenAFs CLIP and / or a Fine-tuned or Domain-specific Models. Examples of current unaligned LLMs may include GPT launched by Open Al, Pathways Language Model (PaLM) developed by Google Al or PaLM 2 LLM that is currently being used for Google’s latest version of Google Bard, XLNet that is an autoregressive Transformer that combines the bidirectional capability of BERT and the autoregressive technology of Transform er-XL to improve the language modeling task, BERT (Bidirectional Encoder Representations from Transformers) which is a deep learning-based technique for NLP developed by Google Brain or LlaMA (Large Language Model Meta Al) launched by Meta.

[0050] For an LLM / LMM aligned with j oumalistic principles, the LLM / LMM 106A may be pre-trained on a large corpus of text or fine-tuned on news articles. For example, the LLM / LMM 106A may be pre-trained on a set of input / output instructions and definitions of journalistic principles for each of the principles described below. The journalistic principles are used to pre-train the LLM in order to detect instances of these in the content but also to be able to generate content that respects these journalistic principles. For example, the journalistic principles may include one or more of: clickbait, subjectivity including hyper partisan, quality of sources with on / Off background source attribution and / or on / Off record source attribution, ad hominem and personal attacks including abusive ad hominem including obscene language, hate speech, circumstantial ad hominem and tuQuoque Ad hominem, red herring and dog whistles. Each of these journalistic principles are well known in the journalistic industry and well understood. Each of these journalistic principles represent content that is antithetical to the journalistic principles and ethics and ideally should be removed from a response that is provided by the LLM / LMM 106A.

[0051] The backend 106 and the system (that includes the trained and aligned LLM / LMM 106A) may include the content scoring system and content scoring 106D (further details of which are disclosed in co-pending and commonly owned US Patent Application Serial No. 18 / 220,437 filed July 11, 2023 that is incorporated herein by reference). The scoring engine 106D may gather pieces of content, such as news pieces of content, from a corpus 300, such as the Internet, with the objective to be able to assess the quality and / or political lean of each piece of content in a programmatic manner by using a set of document quality detectors broken down as a set each for a different journalistic principle. The content scorer may also detect political bias of the article, use domain expertise (trained data joumalist(s)) to teach the system how to score the pieces of content and then use principles from machine teaching, where experts interact with the model, correct its mistakes, iterate so that the machine learning model(s) used to score the pieces of content learns and become better at accurate scoring each piece of content.

[0052] The scoring engine 106D and models therein are designed to emulate the process of a highly trained journalist. The models may be trained on proprietary datasets curated by expert journalists and linguists and utilize vector representations yielded by language models. In one implementation, the one or more models may be transformer-based architectures andrecurrent long-short term memory neural networks that utilize custom attention mechanisms. Attention mechanisms are used to carefully compare the title with the content of the article and detect violations of journalistic principles like clickbait, subjectivity, ad hominem, attacks, quality and type of the sources cited in the article, just as a human expert would do. The one or more models may use different extractive summarization algorithms to enable assessing the degree of relevance of detected violations to the main content of the article and inform the scoring. The one or more models may use a stance detection algorithms to evaluate the stance towards an individual or a topic. Some models may be applied at the sentence level, where a vector representation of each sentence is passed through a neural network model that produces a probability of a violation for that sentence. The sentence level score are collected over all sentences and use different known aggregation algorithms to produce a score over the whole article. The degree of violation of each journalistic principle is used to give a quality score to each article. In one implementation, the final overall scoring model may be a tree-ensemble architecture trained on set of teaching scenarios curated by journalists. The tree-model has learned from the teaching scenarios to adapt to the non-linear dependencies that may exist in news content. For example, subjectivity is expected in certain article types like Op-eds. On the other hand, subjectivity should be penalized heavily in breaking news articles that are straight reporting. Thus, for an LLM / LMM aligned to journalistic principles, the scores of the scoring engine may be used to align the LLM / LMM to the journalistic principles.

[0053] Figure 2 illustrates more details of a training module 106B of the system in Figure 1 that can post-train or fine tune the training of an LLM or LMM to align with different one or more domain specific principles. Each of the modules / elements / engines / devices shown in Figure 2 may be preferably implemented with a plurality of lines of instruct! ons / computer code executed by a processor of the backend 106 so that the processor of the backend 106 may be configured to perform the operations and processes of the training module 106B. Each of the modules / elements / engines / devices shown in Figure 2 also may be implemented in hardware with devices that have a processor executes that plurality of lines of instructions / computer code to perform the operations / processes described below.

[0054] The training module 106B may receive inputs from the vector database 108B, a store of domain principles 202, the web 205 and labeler LLMs or LMMS 204 wherein eachinput may be used to generate instructions for the LLM / LMM training aligned to the domain specific principles. The vector db 108B may contain vector representation of domain principles and may have one or multiple vectors for each principle in addition to vectors for grounding datapoints, each datapoint representing each principle. The store of domain principles 202 may include structured data (json files, RDF format, relational DBs, etc.) and / or unstructured data, such as a task description, a narrative, PDF files, etc.. In one embodiment, the domain principles store 202 may have a plurality of domain principle files (one file for each domain and its principles) wherein each domain principle file is a file describing the principles of the domain and the format of the file may be structured or unstructured.

[0055] The training module 106B may have an instruction generator 106B1 (that may be an instruction generator agent in one embodiment) that selects and generates instructions for domain specific principles that are used to fine tune the training of the LLM / LMM so that the LLM / LMM is aligned with the domain specific principles or multiple sets of different domain specific principles. The instruction generator 106B1 may further include a generator selector 106B1A, one or more retrieval-augmented generator(s) (RAGs) 106B1B, one or more APIbased generator(s) 106B1C, one or more weak labeling generator(s) 106B1D and one or more synthetic instructions generator(s) 106B1E. The generator selector 106B1A may be implemented as a trained machine learning multilabel classification system capable of selecting a subset (one or more) of the instruction generators 106B1B-E. The input to the generator selector 106B1A may be the domain principles 202 and the output is a subset of the appropriate instruction generators that are used to generate the instructions / prompts used to train the LLM / LMM to be aligned with the domain specific principles.

[0056] For example, for the GARM domain principles (example above and principles in Appendix A that is incorporated herein by reference), the generators that are selected may be: 1. Weak labeling generators: generate labeling functions, each checking for a GARM violation and assessing the risk for that violation; 2. Retrieval Augmented Generation (RAG) generator to gather unlabeled data from the web; and 3. Synthetic instruction generator to produce synthetic instructions. For the GARM domain, the weak labeling functions in 1 are used to label the data in 2 and are finally combined with 3 to create a final set of instructions.Retrieval- Augmented Generator

[0057] The retrieval-augmented generators 106B1B may be a search-based system with access to live data on the web 205. The system may include a query generator, a retriever, a ranker, a parser and an instruction generator. The query generator receives, as input, the domain principles and generates one or more web queries. The retriever takes as input the web queries and returns a number of web results relevant to each of the web queries. The ranker ranks the results with respect to their relevance to each of the web queries. The parser then parses the ranked results that may be in html format, to find information that answers each of the web queries. The information is then given to the instruction generator to generate instructions to be used downstream for the fine-tuning of the LLM or LMM.

[0058] For example, using the above already described agriculture example, an retrieval-augmented generator may start with a set of PDFs or e-books on agriculture and the query generator may produce a bunch of questions / queries based on key points of chapters, summaries, tables, graphs etc. within the set of PDFs or ebooks, examples of which are shown in the Table below.Topic QueriesSustainable Agriculture Practices What are sustainable agriculture practices and explain how they contribute to environmental conservation?Agricultural Economics How does the economic viability of organic famiing versus conventional farming in a given region compare to each other?Crop and Soil Sciences What is the role of soil microbiome in crop health and yield. How can farmers enhance soil microbiome diversity9Technology Applications in Agriculture How can precision agriculture technologies contribute to more efficient water usage in farming?

[0059] The retriever of the retrieval -augmentation generator may retrieve a set of enhanced results from the web and the ranker ranks these results with respect to their relevance to each query. A parser of the retrieval-augmentation generator may parse the htmland creates text chunks for each retrieved document and it then filters text chunks that are relevant answers to each query. The pairs of queries and answers are used as instructions to the fine-tuning of a domain-specific LLM using the retrieval augmentation generator.API-Based Generator

[0060] The API-based generators 106B1C may be an API-based system with access to knowledge bases, question and answer API’s, calculator APIs, calendar APIs and other such APIs wherein the APIs may be owned by a third-party or owned by the system. The APIbased system involves a query generator and an instruction generator. The query generator takes, as input, the domain principles and creates requests for the appropriate APIs. The format of the request maybe j son, avro, grpc or any other format that is supported by the client API. Using the same agriculture example above, the API based generator, instead of retrieving documents / answers from the web, the queries are sent directly to a Question and Answer API. The pairs of queries and answers as returned by the Question and Answer API and are used as instructions to the fine-tuning of the domain-specific LLM.Weak Labeling Generator

[0061] The weak labeling generators 106B1D may be a code generating system, capable of generating a set of labeling functions wherein each labeling function is responsible for performing a simple task. The input is a set of domain rules and a collection of unlabeled datapoints that are supplied to the system. The system will internally process each rule and will convert it to a labeling function. An example of a labeling function responsible for detecting whether specific text contains mentions of illegal drugs in the python programming language is the below:def detect illegal drugs text:drug_keywords = ['cocaine', 'weed', 'heroine']text = text.lower()for keyword in drug_key words:if keyword in text:return True # detectedreturn False # Not detected

[0062] Each generated labeling function is the then applied to the unlabeled data to produce an instruction to be used for model training downstream. A simple example of this weak labeling generator is shown in the below table.Generated Input Generated labeling function OutputWhile there is growing def detect_illegal_drugs True for Illegal Drugs evidence supporting the text::effectiveness of medicalmarijuana in treating certain drug_keywords =conditions, there are also ['cocaine', 'weed', 'heroine',risks associated with its use. 'marijuana']Medical marijuana is a text = text.lowerQpsychoactive drug that cancause cognitive and for keyword inpsychological effects, drug_keywords:particularly when used in if keyword in text:high doses or when mixed return True #with other substances. detectedAdditionally, medicalmarijuana can interact with return False #Notother medications, detectedpotentially altering theireffectiveness or causingadverse reactions. Despitethese risks, medicalmarijuana can also providesignificant benefits forcertain conditions, such aschronic pain, nausea andvomiting associated withchemotherapy, and seizuredisorders.These Jews, they are nothing def delect ha te speech t ex t : True for Hatespeech but a bunch of greedy,selfish, money hungry hs_key words = ['jew',people who care nothing for 'niger', 'beaner']anyone but themselves. text = text.lower()They will do and sayanything to get ahead, and for keyword inthey don't care who they hs_key words:step on or hurt in the if keyword in text:process. They are the worst return True #kind of people, and I can't detectedbelieve that we're still lettingthem into our country andGenerated Input Generated labeling function Outputgiving them special return False #Notprivileges. detectedSynthetic Instruction Generator

[0063] The synthetic instruction generators 106B1E are well known and discussed later in this disclosure. Using one or more of these instructions generators, a set of instructions are generated designed to align the LLM / LMM with the domain specific principles.

[0064] The training module 106B may output these set of instructions and have a label aggregator 206 that is a system responsible for taken all outputs from the instruction generators and their outputs and aggregating their responses into a single output. This label aggregation process involves identifying duplicate instructions (from the two or more selected different instructions generators for any particular domain specific principle) that may be associated with different outputs and assigning a single output. The algorithmic process to aggregate the results of aggregators into a single output may involve defining a generative model for how the aggregators generate their labels based on parameters that describe their accuracies and correlations. The parameters of this generative model are then estimated using methods like expectation-maximization (EM) or gradient-based optimization techniques, with or without regularization to prevent overfitting. For example, if there is a dataset consisting of heights and weights of individuals and it is desirable to model this dataset using a GMM to identify clusters corresponding to different body types, the method may use an EM or gradient-based optimization to estimate the parameters of the Gaussian components (mean heights and weights, variances, and covariances) while possibly applying regularization techniques to ensure the model generalizes well and does not overfit the data.

[0065] The output from the label aggregator or label aggregation process 206 (a set of instructions for the domain specific principles with duplicates removed) may be fed into a candidate fine tuning instructions module / process 212 that outputs the candidate fine tuning instructions based on the processes described above. The instructions generator 106B1 may then perform a self-critic process 214 that generates critiques to criticize its own output, in terms of factuality, overall quality, and alignment with the domain principles. The generated critiques will be used to decide whether each generated candidate instruction is sensible inrelation to the task and the domain-specific principles and whether it should be retained or discarded. The results of the self-critic process 214 is a set of final fine tuning instructions database 208. The instructions fine tuning database 208 may contain a set of instructions for various different domain specific principles so that an LLM / LMM can be trained using any one or more of the sets of fine tuned instructions to align the LLM / LMM to one or more domain specific principles for the different domains as discussed above. A training process 210 may train the LLM / LMM using the one or more sets of fine tuned instructions 208 to produce the LLM / LMM aligned to the domain specific principles and thus generate more appropriate responses to a query of a user as discussed above for the GARM, civility and agriculture domains. In this embodiment shown in Figure 2, the LLM / LMM is fine tuned for the domain specific principles for the particular domain. In the embodiment in Figure 2, the LLM / LMM is trained with the instructions to align the LLM / LMM with specific domain principles for a particular domain and the LLM / LMM generates responses aligned with the domain specific principles. In the embodiment shown in Figure 4, an existing LLM / LMM generates a response that is then post-trained so that the existing LLM / LMM generates responses that are aligned with the domain specific principles.

[0066] Figure 3 illustrates a method 300 for fine tuning the training of an LLM or LMM that may be performed by the system shown in Figures 1-2, but may also be performed by another system that has a plurality of lines of computer code / instructions executed by the processor so that the processor is configured to perform the fine tuning of the LLM / LMM and align the LLM / LMM to the principles of the specific domain. In the method, one or more instructions generators are selected (302) wherein those one or more instructions generators each generate a set of instructions that fine tune the LLM / LMM as discussed above. Based on all of the instructions generated by the one or more instruction generators, the method generates a final set of domain principle specific instructions (304) that may be used to train the LLM / LMM (306).

[0067] Figure 4 illustrates a method 400 and data flow for post-training an LLM or LMM to be aligned with domain specific principles. In this alternative embodiment, an already trained LLM / LMM 412 may be post trained (post inference) to align to domain specific principles wherein that LLM / LMM 412 may be owned by a third party of the system. With this post-training process, the same already training LLM / LMM 412 may have its responsesadjusted based on the domain specific principles to which the LLM / LMM 412 is supposed to be aligned. For example, for a prompt from a first user, the LLM / LMM 412 may be posttrained to be aligned the GARM domain principles so that the response to the first user is aligned with the GARM principles. Then, shortly after, a second user may submit a query that requires the LLM / LMM 412 to be aligned with the agriculture domain and, using the post training method 400 shown in Figure 4, the LLM / LMM 412 responses may be adjusted based on the agriculture domain. Thus, in this embodiment, a single already trained LLM / LMM 412 may be re-aligned to different domain specific principles. Note that the method 400 in Figure 4 may be performed using the system in Figures 1-2 although some aspects will be different in this embodiment or may be performed by other systems.Furthermore, each of the processes shown in Figure 4 may be preferably performed by a processor that executes a plurality of lines of instruct ons / computer code so that the processor is configured to perform each of the method processes. In Figure 4, each of the data repositories may be a hardware, software or hardware / software implemented storage for the data, such as a known vector database in one implementation.

[0068] The system that implements the method may have a set of domain specific data 402 that may be retrieved and ingested (404). This domain specific data may be similar to the data discussed above with reference to Figure 2. In one implementation, the domain specific data may be a massive set of news articles that may be rated / scored on clickbait, exaggeration, subjectivity, source quality, dog whistle detection and / or political bias (such as by using the scoring / rating techniques disclosed in co-pending and commonly owned US Patent Application Serial No. 18 / 220,437 filed July 11, 2023 (that is incorporated herein by reference) and that massive set of scored / rated content may be used to post-train an LLM / LMM for the journalistic domain principles. Each different domain may store a different set of domain specific data 402 so that the GARM domain may have its set of domain specific data, while the civility domain or the agriculture domain may each have their own domain specific data. Each set of domain specific data may be used to align the already trained LLM / LMM 412 to be aligned with the principles of the particular domain relevant to the query from a user. The domain specific data 402 may be ingested (by domain alignment data ingestion 406) to generate a domain alignment knowledge graph (KG) 408. The KG may be used to verify, as discussed below, that the LLM / LMM 412 is in alignment with theprinciples of the specific domain that may be determined based on the query posed by the user or may be specified by the user.

[0069] The data in the domain alignment KG (408) may be used by a set of alignment processes 410 to train those alignment processes to be able to adjust a set of candidate responses 413 from the already trained LLM / LMM 412 to generate a response 414 from the already trained LLM / LMM 412 that is aligned with the principles of the particular domain.

[0070] In this method 400, the already trained LLM / LMM 412 may use the vector db 106B, embedding models 418 and retrieved results 422 from the vector db 106B to generate the one or more candidate responses 413 that are likely not aligned to the principles of the specific domain. Thus, the data ingestion process 204 receives the domain specific data and feeds that data into the set of embedding models 218. Each embedding is a representation of values or objects, such as text, images, and audio, that are designed to be consumed by machine learning models / LLMs and translate objects like these into a mathematical form according to the factors or traits each one may or may not have, and the categories they belong to. Thus, the embeddings allows the LLM to find similar objects based on the mathematical form. For example, given a photo or a document, a machine learning model that uses embeddings could find a similar photo or document. Each generated embedding may be stored in the vector db 108B. When a user submits a query / prompt 416, the embeddings are used to search the database 108B for a set of retrieved results 422 that are sent to the LLM / LMM 412 that may then generate the set of candidate responses 413 that are analyzed by the set of alignment processes 410.

[0071] Each alignment process 410 may perform lookups against knowledge graphs of facts and reranking of candidate LLM responses. In one embodiment, the alignment processes 410 may involve synthetic instruction generation using in-depth or in-breadth instruction generation techniques. The techniques may include Retrieval Augmented Generation (RAG) techniques where the training instructions are produced by crawling the web and obtaining live data. Use of external tools and APIs, such as calculators, question and answering systems, and Wikipedia searches. Data programming techniques, where a collection of weak labeling functions in the form of computer code, each representing a specific rule or principle of the domain and used to label unlabeled data points. Weaklabeling functions should not necessarily be human-generated, they could be generated by another LLM, which has been fined-tuned for the specific task of weak label generation. The instruction generators discussed above may also be used. For the post-training embodiment shown in Figure 4, the alignment processes may generate the various instructions as described above and then be fed into the already trained LLM / LMM 412 to adjust the final response 414 to the user query so that the final response 414 is aligned with the principles of the specific domain. Using the process 400 in Figure 4, even an already trained LLM / LMM 412 may be fine tuned to generate responses that are aligned to the principles of the specific domain. In some embodiments, the same selection of the instructions generators and aggregation of the labels from the instructions generators as disclosed in Figure 2 may be used in the post-training process shown in Figure 4. In this post-training embodiment, the already trained LLM / LMM may be supplementally trained / retrained using the set of generated instructions so that the already trained LLM / LMM becomes aligned with the principles of the specific domain pertinent to the query of the user. Thus, the already trained LMM / LLM is retrained using the generated instructions. Once the already trained LLM / LMM is aligned with the principles of the specific one or more domains, it will then be an LLM / LMM, for a future user query, that is already aligned to the principles of the specific one or more domains.AGENTIC WORKFLOW TRAINING OF LLM / LMM ALIGNED WITH DOMAIN SPECIFIC PRINCIPLES

[0072] In an alternative embodiment, the LLM or LMM may be trained and aligned to the domain specific principles using an agentic workflow system as shown in Figures 5A and 5B as described below. Note that the agentic workflow system and method (discussed below with reference to Figures 5A-7) may be used to generate synthetic instructions used to train / align the LLM / LMM as shown in Figure 2 and / or the agentic workflow system and method may generate the synthetic instructions that may be used as part of the alignment processes 410 in Figure 4 for post inference alignment of an already trained LLM / LMM.

[0073] Figure 5A illustrates an agentic workflow system 500 being used to train and align an LLM / LMM to domain specific principles. In one embodiment shown in Figure 5, a first computer system 502 may have a “to be trained” first LLM / LMM 502A (known as astudent LLM / LMM) and a second computer system 504 may have a trainer second LLM / LMM 504A (known as a teacher LLM / LMM). Each LLM / LMM 502A, 504A may be implemented as a plurality of lines of computer code / instructions (executed by a processor of each respective computer system 502, 504) to implement, for example, the transformer used for the LLM / LMM and the LLM / LMM 502A may be trained and aligned to the set of domain specific principles. Note that, in another embodiment, the first and second LLMs / LMMs 502A, 504A may both be hosted or executed on a single computer system. Each computer system may be implemented using cloud computing resources, such as those provided by Amazon web services (AWS), Microsoft Azure, etc., and may include one or more processors, memory, database servers, etc., on which the LLMs / LMMs are implemented.

[0074] Using the system 500 in Figure 5A, the second LLM / LMM (teacher) 504A may be used to generate a set / dataset of synthetic instructions that may be used to train and align the first LLM / LMM (student) 502A or used as part of the alignment process 410 shown in Figure 4 for an already trained LLM / LMM. Each LLM / LMM 502A, 504A may have access to a one or more pieces of domain principles content 506 that may be used to train / align the LLM / LMM 502A and used by the second LLM / LMM 504A to generate the set of synthetic instructions. The one or more pieces of domain principles content 506 may be document / s with domain principles or policies (e.g. a word document, a PDF file, a powerpoint (PPT) file, a list of urls, html etc.) that the student LLM 502A must leam / memorize. The one or more pieces of domain principles content 506 may also include other types of content with domain principles or policies. The teacher LLM / LMM 504A may also have access to one or more teacher resources 508 that may be used by the teacher LLM / LMM 504A to generate the one or more synthetic instructions used to train / align the student LLM / LMM 502A. The one or more teacher resources 508 may include a web search API, a code interpreter and one or more calculators. Details of a method to generate the synthetic instructions / one or more prompts to align an Al model are shown in Figures 6A-7B and described below. Thus, the teacher resources 508 may include the plurality of lines of instructions / computer code executed by a processor of the teacher computer system that implements the method shown in Figures 6A-7B below.

[0075] Using the system 500, the student LLM / LMM 502A may be trained to perform a plurality of tasks / skills using the synthetic instructions generated by the teacher LLM / LMM 504A. The tasks / skills being trained / aligned may include, for example, question and answer (Q&A), summarization, quantitative reasoning, NLP tasks, etc. on the one or more piece(s) of content or parts of one or more document(s). The teacher LLM / LMN 504A may thus generate one or more teaching synthetic instruction(s) to “cover" all the tasks / skills.Examples of the tasks / skills, the synthetic instructions / one or more prompts and the resultant alignment of a response an Al model (LLM or LMM) (the aligned response generated are described below in more detail.

[0076] Figure 5B illustrates more details of the agentic workflow system and process to train and align the LMM / LLM to domain specific principles that uses the same student system 502 and teacher system 504 as described above that have a student model 502B and a teacher model 504B, respectively. Each model 502B, 504B may be implemented as a plurality of lines of instruct! ons / computer code that may be executed by the computer system that hosts the student 502 and teacher 504 to implement the processes and operations of each of the student model and teacher model as discussed below. As shown in Figure 5B, the teacher model 504B may receive principles and produce the synthetic instruction set that may be used to train (see Figure 2 above) or post-train (See Figure 4) above the Artificial intelligence model (“AIM” including an LLM or an LMM) which are then communicated (as acceptable synthetic instructions to the student model 502B). The acceptable synthetic instructions are then used so that the AIM is trained or post-trained to be aligned with the one or more principles of a particular domain. Thus, the teacher 504 may perform agent based knowledge distillation that distills the knowledge (from the one or more pieces of content) into high quality data sets to train AIMs to be aligned to the principles in the knowledge. Thus, the teacher model 504B may generate a plurality of synthetic instructions that may be used to train (such as shown in Figure 2) or post-train with alignment processes (such as shown in Figure 4) the AIM of the student 502 so that one or more responses from the student AIM to a prompt / query from a user are one or more aligned responses and each aligned response is aligned to the one or more principles of the particular domain.

[0077] As shown in Figure 5B, there may also be a known ingestion process 512 that ingests the pieces of content and provides the ingested pieces of content to each of thestudent 502 and the teacher 504. The piece of content may provide the knowledge that is used to align the student AIM based on the knowledge in the pieces of content. Each piece of content that is ingested may be structured or unstructured pieces of content wherein each piece of content may be a knowledge file (policies, PDFs, DB’s, audio files, video files, etc). The pieces of content ingested are available to both the student model and the teacher model wherein the teacher model distills the one or more principles from the knowledge and then transfers the one or more principles to the student model 502B that is now trained and / or post-trained to be aligned with the one or more principles.

[0078] The student model 502B that is trained or post-trained with the one or more principles (in one embodiment in the form of one or more acceptable instructions stored in a principles and task definitions store 510 of the student 502) may include an aligned LLM or LMM model. For example, in one implementation, the student model 502B may be a chatbot agent (or chat agent) whose responses to a user are driven by an LLM that has been aligned with the one or more principles. An example of this chatbot agent may be a chatbot agent that is aligned to one or more principles of particular domains such as the airline industry policies in which the knowledge to generate the one or more principles may include one or more of baggage and optional fees policy, a bags policy, a restricted items policy, a baggage delivery service piece of content, a lost and found policy and a special assistance policy. Another example, is a chatbot aligned with US Food and Drug Administration (FDA) principles so that the system and method disclosed may be used to policy-align a generalist model to a particular section of Title 21 of the United States Code. Title 21 primarily deals with food and drugs. It outlines the federal regulations regarding the manufacture, distribution, and sale of food, drugs, and cosmetics to ensure safety, efficacy, and security. This includes regulations enforced by the FDA (Food and Drug Administration), such as the approval processes for new drugs and the standards for food safety.

[0079] The teacher model 504B may perform a recursive knowledge distillation process by which, using the pieces of content ingested, a plurality of instructions may be generated which may be delivered to the student 502 when acceptable to train or post-train the student LLM / LMM 502A for the one or more principles of the particular domain. The knowledge distillation process may include a generating process 514, a critic process 516 and a routing process 518. The knowledge distillation process may further include a process 520 to requesthuman assistance to perform the knowledge distillation and generate the plurality of instructions. As part of process 520, a human expert may be observing / monitoring the generation of the teacher model and can interrupt / adjust as needed. In addition or alternatively, the system itself may stop and comment “We need a human to intervene here and see if this is correct, or provide more feedback” etc. For example, the original document text may be:The owner or operator of a cosmetic product establishment which is not exempt under § 710.9 and engages in the manufacture or packaging of a cosmetic product is requested to register for each such establishment, whether or not the product enters interstate commerce. This request extends to any foreign cosmetic product establishment whose products are exported for sale in any State as defined in section 201(a)(1) of the act. No registration fee is required.

[0080] The system may have generated a draft QA pair that is:Question: Are all cosmetic product establishments required to register with the FDA?Answer: Cosmetic product establishments are requested to register unless exempt under § 710.9, including foreign establishments exporting to the U.S.

[0081] The system may generate the following message and pause / stop as one example of how human intervention may occur:PAUSE - AMBIGUITY CANNOT BE RESOLVED - request human to help resolve:Does “requested to register” mean registration is mandatory or voluntary under FDA guidelines? Please clarify if this implies a legal requirement or simply a recommendation. :: SYSTEM PAUSES::

[0082] Using the pieces of content ingested, the generation process 514 may generate a first plurality of instructions for one or more principles of the particular domain as described below in more detail. A sample of these first plurality of instructions may be input to the critic process 516 that programmatically performs a critique on each instruction and perform refinement of the QA pair. For example, a summary may be refined across sub-trees, improving its quality iteratively. The high-level document summary may be:The document outlines the voluntary registration process for cosmetic product establishments, detailing who should register, how to register, the information required, and exemptions.

[0083] The system may perform a phase 1 refinement so that the summary becomes:The document explains the voluntary registration process for cosmetic product establishments, including who should register (§ 710.1), the 30-day registration timeline (§ 710.2), and how to obtain and submit the registration form (§ 710.3). It specifies the required information, such as the establishment’s name, address, business type (§ 710.4), and the need to update or cancel registrations if circumstances change (§ 710.5). Exemptions apply to certain professionals and organizations (§ 710.9).

[0084] The system then may perform a phase 2 refinement so that the summary becomes:The document governs the voluntary registration of cosmetic product establishments, emphasizing the request for owners or operators to register, including foreign entities exporting to the U.S. (§ 710.1). Registration must occur within 30 days of commencing operations (§ 710.2) by submitting Form FD- 2511, available from the FDA, which requires detailed information about the establishment (§ 710.3, § 710.4). Amendments to registration must be submitted within 30 days of any changes (§ 710.5), and the FDA assigns a permanent registration number (§ 710.6). Misrepresentation of registration status is considered misleading (§ 710.8). Exemptions cover specific categories like beauty shops, medical institutions, and those solely manufacturing for research (§ 710.9).

[0085] Thus, the system may iteratively refine the summaries and thus the other processes to generate better alignment data to train or post-train the student model.

[0086] The critic 516 may determine that each instruction aligns with the one or more principles of the particular domain (it is acceptable) or that the instruction does not align with the one or more principles of the particular domain (it is not acceptable). In other words, any generated data (QA pair, for example) that does not need further refinement (an example of which is set forth above) may be “acceptable.”

[0087] The routing process 518 may then route the acceptable instructions onto the student model 502B and route the unacceptable instructions back to the generation process 514 for adjustment. Using this recursive knowledge distillation process, the teacher model 504B uses the pieces of content for the particular domain to generate the plurality of instructions that may be used to train or post-train the student model to be aligned to the one or more principles of the particular domain. As mentioned above, the recursive knowledge distillation process includes the human intervention process 520 as needed.Multi-Stage Agentic Workflow for Domain-Specific LLM Alignment

[0088] Using the agentic system architecture of Figures 5 A and 5B discussed above or another agentic system architecture, a multi-stage agentic workflow process may be performed to generate the prompts / instructions that may be used to align the artificial intelligence model to a specific domain. In other words, the process / workflow is a methodology for aligning artificial intelligence models (LLMs or LMMs) to learn domainspecific principles through a multi-stage process of document processing, knowledge extraction, and synthetic instruction generation of training data. Similarly, parts of the agentic system and workflow that are used for generating the data for fine-tuning can be used as guardrail s / sy stem prompts / chain-of-thought / iterative self-refinement to force the model to think harder before giving an aligned answer.

[0089] Figure 6 illustrates an agentic workflow method 600 for generating a synthetic dataset of instructions. In one implementation, the process 600 in Figure 6 may be performed by the computer system 504 that hosts / executes the teacher LLM / LMM 504A and may be performed using the teacher resources 508 in Figure 5. However, the method 600 may be performed using other computer system resources and architectures. The method 600 may be performed for each piece of content (document, podcast, blog, audio, etc.) that may be used for training data to align an Al model (LLM or LMM) to the domain desired by the user, such as the domain examples set forth above.

[0090] In the method, a structured representation of each piece of content (602) of the domain principles content 508 is created as part of a recursive document summarization process 604. The recursive document summarization may be a first stage of the multi-stage agentic workflow method. The purpose of this first stage is to recursively generatehierarchical summaries of document content that preserve context and relationships between different sections (where each node in the hierarchical document tree corresponds to a document section), based on the agentic system instructions (that is, prompts to an instruction-tuned Al base model).

[0091] In one embodiment, the structured representation of each piece of content may be a hierarchical document tree. The structured representation of each piece of content may organize the words, phrases, sections, etc. of each piece of content into a well-known document tree wherein the document tree has one or more nodes and each node in the tree can be connected to many children (depending on the type of tree), but must be connected to exactly one parent, except for the root node, which has no parent (i.e., the root node as the top-most node in the tree hierarchy). A node that does not have a child may be known as a leaf node.

[0092] The hierarchical document tree, for each document, may be constructed for each document (See Figure 6B, tree A for example that shows the label and (if available) extracted text content in each node.) Each node either contains text content or acts as a structural element; that is, some document sections, or nodes, might contain a header label but lack text, such as a high-level document header (e.g. the document’s title). As shown in Figure 6B tree A, each node may be tagged with labels and hierarchical relationships (that is, which document sections, or nodes, are parents or children of other document sections). Appendix C contains an example of a document tree. In this example, the document tree is a customer service plan document for an airlines that shows the labels, content and nodes for the document tree including a root node at the end of Appendix C.

[0093] The hierarchical document tree formation may also include leaf Node Processing in which any leaf nodes (nodes in the tree without children) are identified. For each leaf node that has / contains content, the leaf node processing may applyLEAF SUMMARIZATION PROMPT (See Table X below) to generate concise yet informative summary for each leaf node. For each leaf node with content, the lead node processing may maintain context through "breadcrumbs" showing path to root and include a Records summary in node's attributes (See Figure 6B tree B).

[0094] The hierarchical document tree formation may also include parent node synthesis in which the method processes non-leaf nodes (each is a parent node) in post-order traversal. For each parent node, the parent node synthesis process collects summaries from all child nodes and applies a NODE SUMMARIZATION PROMPT (See Table X below) to synthesize combined summary. The parent node synthesis may also create cohesive narrative incorporating key points from children and store synthesized summary in node's attributes. Note that some nodes might not contain any text content, such as a high-level document header (e.g. the document’s title), but they are assigned summaries based on a synthesis of their children’s summaries (which themselves are based on the recursively generated summaries of their children, and so on, starting from the document’s tree leaf nodes) For example, as shown in Figure 6B tree A, the root node, R, has no text, but the parent node synthesis may, for root node R, recursively generate a summary despite the lack of text at the root node level as shown in Figure 6B tree B “Summary” that appears in root node R.Table XPrompts for Recursive Summarization of the Content of a Hierarchical Document Tree (Precursor to Synthetic Data Generation)Prompt ExampleLeaf Summarization Prompt You are a document summarizes You will be (LEAF SUMMARIZATION PROMPT) given the name of the current section and the location of the current section you are summarizing in the document using breadcrumbs. You must summarize the content of the current section and produce a single narrative summary. It is very important that you do not include / inject any new information. You must only use the information in the text provided to you. Note that if the text is very short, it's okay if your generated summary closely resembles the original text.Section name: {label}Section breadcrumbs: {breadcrumbs} Section content:Prompt Example{content}Parent (Non-Leaf) Node Summarization You are a document summarizes You will be Prompt given:(NODE SUMMARIZATION PROMPT) 1. the name of the current section2. the location of the current section you are summarizing in the document using breadcrumbs3. summaries of this sections several subsectionsYou must combine and synthesize these summaries into a single narrative summary of the entire section that describes what the section is about, and captures its main points, without just listing out the sub-section summaries. You must condense the information to give a high level overview of the section, and its main points.It is very important that you do not include / inject any new information. You must only use the information in the text provided to you. Remember you must summarizeSection name: {label}Section breadcrumbs: {breadcrumbs} Section content:{content}

[0095] Returning to Figure 6A, the recursive summarization process 604 is performed. Figure 6B tree B shows an example of the recursive summarization process (See summary in each node of tree B) even when a node (such as the root node R) does not have any text. Using the recursive summary process, a summary for the root node R is generated from the text and summaries of the other nodes in the document tree. The method may create summaries of increasing extractiveness (lots of details, facts, data, etc.), as it recurses the structured representation (for example, the document tree). As a result, "leaf1sections (thatis, sections without any subsection or "children") will be summarized much more exactively than their "ancestors" sections.

[0096] Once the recursive summarization is completed, the method may generate question / answer (QA) pairs (606) based on the summaries in the structured representation of the document. Figure 6B tree C shows an example of the structured representation of the document with the QA pairs for each node. The purpose of the QA pair generation is to create training messages data for the subsequent fine-tuning of an instruction-tuned Al model (LLM and / or LMM). The QA pairs may be synthetically generated so that the QA pairs comprehensively probe the input document's knowledge, based on the agentic system instructions.

[0097] Table AA below contains an example of a section of a document on which the recursive summarization process 604 is performed to generate a summary (See middle column of Table AA) and the QA pair generation process 606 is performed to generate QA pairs (see right column in Table AA) for each section of the document.TABLE AAOriginal Section Text Recursively Generated GeneratedSummary Question+ Answer Pairs § 710.1 Who should Cosmetic product • Question: Whois register. establishments involved in required to register a manufacturing or packaging cosmetic product cosmetics, whether or not establishment? The owner or operator of atheir products enter Answer: Owners or cosmetic productinterstate commerce, are operators of establishment which is notrequested to register unless cosmetic product exempt under § 710.9 andexempt under § 710.9, establishments engages in the manufactureincluding foreign engaged in or packaging of a cosmeticestablishments exporting manufacturing or product is requested toproducts to the U.S. packaging cosmetics, register for each suchRegistration should occur including foreign establishment, whether orOriginal Section Text Recursively Generated GeneratedSummary Question+ Answer Pairs not the product enters within 30 days of starting establishments interstate commerce. This operations using Form FD- exporting to the request extends to any 2511, obtainable from the U.S., are requested foreign cosmetic product FDA at 5001 Campus Dr., to register unless establishment whose College Park, MD, or exempt under products are exported for district offices. The form, specific provisions. sale in any State as defined which requires details such • Question: When in section 201(a)(1) of the as the establishment’s name, should a cosmetic act. No registration fee is address, business trading product required. names, and type of business establishment (manufacturer and / or register with the packer), must be mailed to FDA?§ 710.2 Time forthe same FDA address. No Answer:registration.fees are required for Registration should registration. be completed within The owner or operator of an 30 days after the establishment entering into establishment begins the manufacture or manufacturing or packaging of a cosmetic packaging cosmetic product should register his products. establishment within 30 • Question: How can days after the operation an establishment begins. obtain the necessary form for registration? § 710.3 How and where toAnswer: The register.registration form (Form FD-2511) canOriginal Section Text Recursively Generated GeneratedSummary Question+ Answer Pairs Form FD-2511 be requested from (“Registration of Cosmetic the FDA’ s office at Product Establishment”) is 5001 Campus Dr., obtainable on request from College Park, MD, the Food and Drug or from any FDA Administration, 5001 district office. Campus Dr., College Park, • Question: What MD 20740, or at any Food details must be and Drug Administration provided when district office. The registering a completed form should be cosmetic product mailed to Cosmetic Product establishment? Establishment Registration, Answer: The Food and Drug registration form Administration, 5001 requires the Campus Dr., College Park, establishment’s MD 20740. name, address with ZIP code, all business trading [39 FR 10059, Mar. 15,names, and whether 1974, as amended at 68 FRit operates as a 15355, Mar. 31, 2003; 81manufacturer, FR 49897, July 29, 2016]packer, or both.• Question: Is there a § 710.4 Information fee associated with requested. registering a cosmetic product establishment?Original Section Text Recursively Generated GeneratedSummary Question+ Answer Pairs Form FD-2511 requests Answer: No, there is information on the name and no fee required for address of the cosmetic registering a product establishment, cosmetic product including post office ZIP establishment with code; all business trading the FDA. names used by the • Question: Where establishment; and the type should the completed of business (manufacturer registration form be and / or packer). The sent? information requested Answer: Completed should be given separately registration forms for each establishment as should be mailed to defined in § 700.3 (j) of this the FDA at Cosmetic chapter. Product Establishment Registration, 5001 Campus Dr., College Park, MD 20740.

[0098] Figures 7A and 7B illustrate further details of the method 606 for QA pair generation. The method may first perform a relevance assessment process 702 in which the computer system executing the method 606 may evaluate each node's summary field’s relevance to the user’s provided instructions as to how the final Al model (LLM or LMM) should be fine-tuned. Table Y below shows examples of relevance prompts (INSTRUCTIONS_RELEVANCE_PROMPT ) that may be generated and used to make binary decisions on the nodes as to their relevance. The process 702 may also mark nodes for QA generation based on relevance.

[0099] Once the relevant nodes (to the user’s instructions to align the Al model (LLM or LMM) are identified, the method may perform an initial QA generation process 704 for each relevant node. The initial QA generation process, for each relevant node, may apply a GENERATION ! ASK PROMPT (an example of which is shown in Table Y) to create QA pairs in reference to the user’s provided instructions. The method 704 may consider a node's hierarchical level to determine question complexity and generate a specified number of unique QA pairs for this relevant node. The method may then determine if there are more relevant nodes to process (705) and loop back to process 704 to generate the initial QA pairs for the other relevant nodes. If there are no more relevant nodes, the method moves to an answer refinement process 706 that is shown in Figure 7B.

[0100] During the answer refinement process 706, the computing system that performs the answer refinement may determine if each QA pair needs assessment. For example, the method may evaluate if child nodes of a parent node contain information to improve the parent node’s QA pair answers. This process may use a NEED REFINE PROMPT (See example in Table Y) to identify relevant children / child nodes and creates list of children indices for refinement. The answer refinement process 706, for each identified child node, may apply a REFINE TASK PROMPT (See example in Table Y) to incorporate additional context. The answer refinement process maintains conciseness while improving accuracy, updates QA pairs with refined answers and may record QA pairs list in node's attributes (an example is shown in Figure 6B tree C).

[0101] The method 606 may then perform a summary QA addition process 708 that appends additional QA pair using node's summary (effectively, “What is this section about?”), creates comprehensive overview question for each section and ensures coverage of high-level concepts.

[0102] Once the QA pairs are generated, the method 600 may perform a system prompt generation process (608) whose purpose is to create a specialized system prompt that will guide the fine-tuned model's behavior and responses in alignment with specific domain requirements of the user. To perform the system prompt generation, the method performs input processing in which the method collects document summary from root node, processes user-provided instructions and context and validates input completeness. Once inputprocessing is completed, the process 608 may perform the prompt generation, such as by using a GEN SYSTEM PROMPT PROMPT (an example of which is shown in Table Y) to create one or more specialized prompts that incorporates document context and user instructions, establishes model identity and purpose and / or establishes scope of questions and answers to be fielded by the model.Table YPrompts for Synthetic Generation of the Training Data Messages for Model Fine-tuningRelevance Prompt Example ResultsUser Instructions Relevance Prompt Your task is to judge whether a given text is (INSTRUCTIONS RELEVANCE PROMPT) relevant to a set of instructions. You will provide a Boolean response indicating whether the text is relevant or not.To complete your task, you are provided with:(1) the text to be judged(2) the document header of the text (3) the instructions to compare against Important points to consider:(1) The relevance should be judged based on whether the text contains information that could be useful in following or understanding the given instructions.(2) The text doesn't need to fully address the instructions to be considered relevant; partial relevance is sufficient.(3) Consider both explicit and implicit relevance. Sometimes the connection might not be immediately obvious but could still be important.(4) If you're unsure, err on the side of marking the text as relevant.The information for your task:1. text: {text}2. document header of the text:{text_header}3. instructions: {instructions}Relevance Prompt Example ResultsYour output should be a JSON object with a single key: 'relevant', and a boolean value (true or false).Example output:"'json{{"relevant" : true}}Or:"'json{{"relevant": falseProvide only the JSON object in your response; do not include any additional characters before or after the JSON object, including any commentary.QA Pair Generation Prompt Your task is to generate {num_questions} (GENERATION TASK PROMPT) unique question-answer pair(s) about a given text, such that the question can be either fully or partially answered by consulting the content of the text.To complete your task you are provided with:(1) the main text(2) its document header(3) its hierarchical level / location in the document(4) a list of all possible hierarchical levels in the text in decreasing order of generality (5) instructionsNote the below important points:(1) If the text has a higher hierarchical level, in general try to bias your generated question to be a "why / how" kind ofRelevance Prompt Example Resultsquestion (these questions are usually more complex), and if the text has a lower hierarchical level, bias your question (though not always) to be a "what / when / where / who" kind of question (i.e. a question with a straightforward descriptive answer).(2) Your question cannot contain vague references to the text's header, such as 'this section'; any references to the text's header must explicitly state the header name. (3) Your question must include sufficient context; e.g. a question such as 'What happens if a request for confidentiality lacks sufficient data?' contains insufficient context - here, either the text's header name should be explicitly included in the question, or the context of the question could instead be provided through enhanced descriptive language -- in the foregoing example, the question should be expanded to describe what 'what happens' means and what 'request for confidentiality' refers to (what is being requested?).(4) Both the question and answer must be relevant to the provided instructions.Generate questions that probe aspects of the text pertinent to the instructions, and provide answers that are useful in the context of the instructions.(5) Your output should consist of a list containing {num_questions}-many JSON objects - each with two keys: 'question' and ' answer' .Find below the information relevant to your task:1. text: {text},2. document header of text: {text_header}, 3. hierarchical level of text: {textjevel}, 4. possible levels in decreasing hierarchical order: ['O’, ’1’, '2', '3', '4', '5', '6', ...],5. instructions: {instructions}Relevance Prompt Example ResultsIn your output, do not include any additional characters before or after the list of JSON objects, including any commentary.QA Pair Need-Refinement Prompt Your task is to decide if a given generated (NEED REFINE PROMPT) question-answer pair from a main text could use refinement from any of the children of the node from which the question-answer pair was generated.To complete your task you are provided with:(1) the main text of a given node,(2) the generated question(3) the generated answer(4) a dictionary of index:title of the children of the node from which the question-answer pair was generatedYour response should consist **only** of a list of the **indexes** of the children whose text would enhance the answer to the question by providing more context or critical information missing in the main text, etc.If you do not think any children should be consulted, return an empty list: '[]'Find below the information relevant to your task:1. main text: {text},2. generated question: {question},3. generated answer: {answer},4. children: {children}In your output, do not include any additional characters before or after the list of indexes, including any commentary. QA Pair Refinement Prompt Your task is to refine / improve on the (REFINE TASK PROMPT) current answer to the original question by consulting the additional text provided. If the additional text contains information that would enhance the answer to the originalRelevance Prompt Example Resultsquestion, you should provide a refined answer. If the additional text does not contain any information that would enhance the original answer to the question, you should just return the original answer. Important: Your refined answer should NOT just simply incorporate the additional text! You must still provide a short, concise answer to the original question. The additional text should only help you refine that answer - not elongate it.Important: Do not include any additional information in your output that is not found in the original question, current answer, or additional text.Find below the information relevant to your task:1. original question: {question}2. current answer: {answer}3. additional text: {additional text} Your output should consist ONLY of a JSON object with two keys 'useful': true / false (lowercase) - whether the additional text was useful in refining the answer, and 'new_answer': the refined answer if the additional text was useful, or the original answer if the additional text was not useful.In your output, do not include any additional characters before or after the JSON object, including any commentary. Model System Prompt Generation Prompt Your task is to generate a SYSTEM (GEN SYSTEM PROMPT PROMPT) PROMPT for a custom LLM model. The custom LLM model is being trained to discuss various aspects of a document with a user. You are not being provided with the whole document, just a high level overview of it.Relevance Prompt Example ResultsYou are also being provided with a set of instructions / context that the user has given, expressing their intentions for the custom LLM. These are provided in a conversational format, and you should consider them when generating the system prompt.You must use this document summary and the instructions / context provided by the user to generate a suitable SYSTEM PROMPT for the custom LLM. Recall - a SYSTEM PROMPT is the text that an LLM uses to ground each of its responses. It should establish a basic identity, purpose and scope for the LLM. It is worded as if speaking directly to the LLM - like "You are a chatbot that answers..." etc.You are provided with the following information:Document Summary:{ document_summary } Instructions / Context:{instructions}Your response should contain a single JSON object with a single key "system_prompt" and a string value containing the system prompt you generate. Include no other commentary before or after.

[0103] Figure 8 illustrates a method 800 for aligning an LLM / LMM 502A / 502B using the synthetic dataset of instructions generated by the teacher LLM / LMM 504A / 504B. In the method, the teacher LLM / LMM 504A / 504B has generated the synthetic set of instructions for the particular domain as described above and shown in Figures 6A-7 and the accepted synthetic set of instructions for the domain (to align the student with the skills and knowledge relevant to the particular domain) are communicated to the student model / LLM / LMM 502B (802) and stored in the principles and task definitions 510 of the student LLM / LMM 502A. The accepted synthetic set of instructions for the domain are used (804) to train the LLM / LMM 502B to align it with the skills and knowledge of the particular domain. The aligned LLM / LMM 502B may then (806) respond to prompts / queries from a user that is consistent with the skills and knowledge for the particular domain. Thus, the methods in Figure 6 and 7 achieve the technical goal of aligning an LLM / LMM to a set of skills and knowledge for a particular domain. Furthermore, as described above, the LLM / LMM 502A may be trained to be aligned with the one skills and knowledge of multiple domains as well.

[0104] The foregoing description, for purpose of explanation, has been with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The embodiments were chosen and described in order to best explain the principles of the disclosure and its practical applications, to thereby enable others skilled in the art to best utilize the disclosure and various embodiments with various modifications as are suited to the particular use contemplated.

[0105] The system and method disclosed herein may be implemented via one or more components, systems, servers, appliances, other subcomponents, or distributed between such elements. When implemented as a system, such systems may include and / or involve, inter alia, components such as software modules, general-purpose CPU, RAM, etc. found in general-purpose computers. In implementations where the innovations reside on a server, such a server may include or involve components such as CPU, RAM, etc., such as those found in general-purpose computers.

[0106] Additionally, the system and method herein may be achieved via implementations with disparate or entirely different software, hardware and / or firmware components, beyond that set forth above. With regard to such other components (e.g., software, processing components, etc.) and / or computer-readable media associated with or embodying the present inventions, for example, aspects of the innovations herein may be implemented consistent with numerous general purpose or special purpose computing systems or configurations.Various exemplary computing systems, environments, and / or configurations that may be suitable for use with the innovations herein may include, but are not limited to: software or other components within or embodied on personal computers, servers or server computing devices such as routing / connectivity components, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, consumer electronic devices, network PCs, other existing computer platforms, distributed computing environments that include one or more of the above systems or devices, etc.

[0107] In some instances, aspects of the system and method may be achieved via or performed by logic and / or logic instructions including program modules, executed in association with such components or circuitry, for example. In general, program modules may include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular instructions herein. The inventions may also be practiced in the context of distributed software, computer, or circuit settings where circuitry is connected via communication buses, circuitry or links. In distributed settings, control / instructions may occur from both local and remote computer storage media including memory storage devices.

[0108] The software, circuitry and components herein may also include and / or utilize one or more type of computer readable media. Computer readable media can be any available media that is resident on, associable with, or can be accessed by such circuits and / or computing components. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and can accessed by computing component. Communication media may comprise computer readable instructions, data structures, program modules and / or other components. Further, communication media may include wired media such as a wired network or direct-wired connection, however no media of any such type herein includestransitory media. Combinations of the any of the above are also included within the scope of computer readable media.

[0109] In the present description, the terms component, module, device, etc. may refer to any type of logical or functional software elements, circuits, blocks and / or processes that may be implemented in a variety of ways. For example, the functions of various circuits and / or blocks can be combined with one another into any other number of modules. Each module may even be implemented as a software program stored on a tangible memory (e g., random access memory, read only memory, CD-ROM memory, hard disk drive, etc.) to be read by a central processing unit to implement the functions of the innovations herein. Or, the modules can comprise programming instructions transmitted to a general-purpose computer or to processing / graphics hardware via a transmission carrier wave. Also, the modules can be implemented as hardware logic circuitry implementing the functions encompassed by the innovations herein. Finally, the modules can be implemented using special purpose instructions (SIMD instructions), field programmable logic arrays or any mix thereof which provides the desired level performance and cost.

[0110] As disclosed herein, features consistent with the disclosure may be implemented via computer-hardware, software, and / or firmware. For example, the systems and methods disclosed herein may be embodied in various forms including, for example, a data processor, such as a computer that also includes a database, digital electronic circuitry, firmware, software, or in combinations of them. Further, while some of the disclosed implementations describe specific hardware components, systems and methods consistent with the innovations herein may be implemented with any combination of hardware, software and / or firmware. Moreover, the above-noted features and other aspects and principles of the innovations herein may be implemented in various environments. Such environments and related applications may be specially constructed for performing the various routines, processes and / or operations according to the invention or they may include a general -purpose computer or computing platform selectively activated or reconfigured by code to provide the necessary functionality. The processes disclosed herein are not inherently related to any particular computer, network, architecture, environment, or other apparatus, and may be implemented by a suitable combination of hardware, software, and / or firmware. For example, various general-purpose machines may be used with programs written in accordance with teachingsof the invention, or it may be more convenient to construct a specialized apparatus or system to perform the required methods and techniques.

[0111] Aspects of the method and system described herein, such as the logic, may also be implemented as functionality programmed into any of a variety of circuitry, including programmable logic devices ("PLDs"), such as field programmable gate arrays ("FPGAs"), programmable array logic ("PAL") devices, electrically programmable logic and memory devices and standard cell-based devices, as well as application specific integrated circuits. Some other possibilities for implementing aspects include: memory devices, microcontrollers with memory (such as EEPROM), embedded microprocessors, firmware, software, etc. Furthermore, aspects may be embodied in microprocessors having software-based circuit emulation, discrete logic (sequential and combinatorial), custom devices, fuzzy (neural) logic, quantum devices, and hybrids of any of the above device types. The underlying device technologies may be provided in a variety of component types, e.g., metal-oxide semiconductor field-effect transistor ("MOSFET") technologies like complementary metal-oxide semiconductor ("CMOS"), bipolar technologies like emitter-coupled logic ("ECL"), polymer technologies (e.g., silicon-conjugated polymer and metal-conjugated polymer-metal structures), mixed analog and digital, and so on.

[0112] It should also be noted that the various logic and / or functions disclosed herein may be enabled using any number of combinations of hardware, firmware, and / or as data and / or instructions embodied in various machine-readable or computer-readable media, in terms of their behavioral, register transfer, logic component, and / or other characteristics. Computer-readable media in which such formatted data and / or instructions may be embodied include, but are not limited to, non-volatile storage media in various forms (e.g., optical, magnetic or semiconductor storage media) though again does not include transitory media. Unless the context clearly requires otherwise, throughout the description, the words "comprise," "comprising," and the like are to be construed in an inclusive sense as opposed to an exclusive or exhaustive sense; that is to say, in a sense of "including, but not limited to." Words using the singular or plural number also include the plural or singular number respectively. Additionally, the words "herein," "hereunder," "above," "below," and words of similar import refer to this application as a whole and not to any particular portions of this application. When the word "or" is used in reference to a list of two or more items, that wordcovers all of the following interpretations of the word: any of the items in the list, all of the items in the list and any combination of the items in the list.

[0113] Although certain presently preferred implementations of the invention have been specifically described herein, it will be apparent to those skilled in the art to which the invention pertains that variations and modifications of the various implementations shown and described herein may be made without departing from the spirit and scope of the invention. Accordingly, it is intended that the invention be limited only to the extent required by the applicable rules of law.

[0114] While the foregoing has been with reference to a particular embodiment of the disclosure, it will be appreciated by those skilled in the art that changes in this embodiment may be made without departing from the principles and spirit of the disclosure, the scope of which is defined by the appended claims.APPENDIX AGlobal Alliance for Responsible Media (GARM) PrinciplesGARM standards are designed to determine presence of sensitive topics in content, alongwith a associated risk level.Category & Risk DefinitionsThe following definitions are GARM’s category and risk level definitionsCategory: The type of risky content a particular text may fall under.Risk: The perceived level of risk to advertisers based on GARM standards.The levels from least to most risk are None / No Risk, Low, Medium, High, and Floor. The “Floor” level consists of the worst content for each category and is generally considered inappropriate for any advertising.Adult & Explicit • Promotion of Illegal sale, distribution, and consumption of child porSexual Content nography• Explicit or gratuitous description of sexual acts, and / or genitals Arms & • Promotion and advocacy of sales of illegal arms, rifles, and handguns Ammunition • Instructive content on how to obtain, make, distribute, or use illegal arms• Glamorization of illegal arms for the purpose of harm to others• Promotion of use of illegal arms in unregulated environments Crime & Harmful • Graphic promotion, advocacy, and depiction of willful harm and actual acts to individuals unlawful criminal activity. Explicit violations / demeaning offenses of and Society, Human Human Rights (e.g. human trafficking, slavery, self-harm, animal cruRight Violations elty etc.)• Harassment or bullying of individuals and groupsDeath, Injury or • Promotion, incitement or advocacy of violence, death or injury Military Conflict • Vivid depictions of willful harm to others• Incendiary content provoking, enticing, or evoking military aggression • Live action footage / photos of military actions & genocide or other war Online piracy • Promotion or advocacy of pirating, copyright infringement & counterfeiting• Instructional content on pirating, copyright infringement & counterfeiting for malicious purposeHate speech & acts • Speech that incites hatred, promotes violence, vilifies, or dehumanizes of aggression groups or individuals based on race, ethnicity, gender, sexual orientation, gender identity, age, ability, nationality, religion, caste, victimsand survivors of violent acts and their kin, immigration status, or serious disease sufferers.Obscenity and • Excessive use of profane language or gestures and other repulsive acProfanity, including tions that shock, offend, or insultlanguage, gestures,and explicitly gory,graphic or repulsivecontent intended toshock and disgustIllegal • Promotion or sale of illegal drug use, including prescription drug Drugs / T obacco / ecigabuse. Federal jurisdiction applies, but allowable if legal local jurisarettes / diction can be determined (ex: permissible if discussing legal usage of Vaping / Alcohol cannabis in California)• Promotion and advocacy of tobacco, e-cigarette (Vaping), and alcohol useSpam or Similarly • Promotion or advocacy of malware / phishingHarmful Content • Instructional content on malware / phishing for malicious purposes Terrorism • Promotion and advocacy of graphic terrorist activity involving defamation, physical and / or emotional harm of individuals, communities, and societyDebated Sensitive • Insensitive, irresponsible and harmful treatment of debated social isSocial Issue sues and related acts that demean a particular group or incite greaterconflictAPPENDIX BCivility PrinciplesCivility in any content is comprised by 2 attack types: severe attack and general attack. See definitions below for each attack type.Definition of AttackInsulting language toward an individual or group is considered as an attack. This includes but is not limited to:• Name-calling• Derogatory terms• Paraphrasing / quoting insults• Insinuating something insulting• Mocking through imitation• Threats / threatening language• Self-deprecation, but only if using offensive languageo use dictionary flags as a guide for determining offensivenessSevere AttackAn attack is severe when it is based on characteristics with legal protections against discrimination. For instance, insulting someone based purely on their race is considered a severe attack.The list of legally-protected characteristics varies depending on location, but we account for the characteristics protected by US federal law:• Race / color• Religion• National origin• Age (40 and over)• Sex• Gender identity (incl. respecting someone’s expressed I preferred gender identity) • Sexual orientation• Pregnancy• Familial status• Disability status• Veteran status• Genetic informationGeneral AttackGeneral attacks form a much broader category and are trickier to describe. They must meet all the requirements for attack above, but must not cross the line into attacking protected characteristics like severe attacks.To better understand what general attacks can look like, please see the Examples section below.Not an AttackThe following are cases that may be confused with attacks, but should not by default be classified as such.• Reporting or statement of the following without insulting language:o negative review or sentiment of the subject in discussiono unflattering or biased reporting that put the subject in a bad lighto allegation or personal assessment that is unfavorable to the subject • Offensive language, expletive, or profanity that is not used to insult a person or group o includes titles / names of creative works• Discussion of sensitive topics like race, gender, sexuality, etc. without any insults or offensive languageExamplesAttack Type Example ExplanationGeneral He’s the definition of spoiled Attacks someone’ s character.entitlementGeneral Uh, no, not Steve Bannon, but Disparages appearance and we'll get it to the right one. isn’t tied to a protected There we go. Blackstone, characteristic.look at that ugly face.Severe I do not trust any Jew. I do not Expresses negativity toward respect Jews. Uh, these are and vilifies a race, ethnicity, our enemy. They're trying to and religion.wipe out EuropeansSevere F**kyouFag Slur based on (perceived) sexual orientation.Not an attack She thinks she's just going Contains profanity that can be back to this guy's apartment used in a friendly or insulting that she's really hit it off with. way. In this case, the word is I mean, I speak for myself, used as the name for a party how many fucking times have and is not meant to be an I done that? insult.Not an attack Right, that's great for you. It Expresses dislike / negative doesn't work for me. I can't preference without beingyou can't convince me to like insulting.her. I don't like herNot an attack There's so many different Engaging in self-deprecation things, your engineering skills and suggesting not being very for some of you. I can't even skilled or capable of a basic change a light bulb, right? task. No offensive language isused.APPENDIX CDocument Tree for: Airlines Customer Service Plan {"children": [{"children": ["content": "Customer service plan This content can be expanded --customerserviceplan American Airlines and American Eagle are in business to provide safe, dependable and friendly air transportation to our customers, along with numerous related services. We are dedicated to making every flight you take with us something special. Your safety, comfort and convenience are our most important concerns. American Airlines and American Eagle initially submitted their joint Customer Service Plan to the Department of Transportation (DOT) in 1999 and updated it in 2010 and 2011. American Airlines and its regional airline partner American Eagle serve almost 250 cities around the world, from Abilene to Zurich, and operate more than 3400 daily flights. Our combined fleet numbers more than 900 aircraft. In addition to our extensive domestic service, we serve numerous destinations in Europe, Asia, the Caribbean, Canada, Mexico, and Latin America. The American frequent flyer program, AAdvantage\u00ae, is the oldest and largest in the world. We are in business to provide safe, dependable, and friendly air transportation to our customers, along with numerous related services, in the hopes that you will fly us again and again. We work very hard to make your entire experience with us, from making a reservation to deplaning at your final destination, a positive one. Although we are successful in this effort most of the time, there are times when things do not go as smoothly as we, and you, would like. Operating a network of more than 3400 flights and servicing hundreds of thousands of passengers each day is challenging and complex. Inevitably, some of our flights are affected by adverse circumstances, some of which are within our control and some of which are not. This Customer service plan addresses a number of the service goals we have defined for ourselves. Every customer is important to American Airlines and American Eagle, and we are dedicated to making every flight you take with us something special. Your safety, comfort and convenience are our most important concerns. * Accommodation of customers with special needs * Assistance when your flight has been delayed or canceled * Baggage delivery * Baggage liability * Check-in requirements and baggage acceptance cutoff times * Customer loyalty program \u2013 AAdvantage\u00ae * Delays, cancellations and diversion events * Essential customer needs during extraordinary delays * Family seating * Flights with oversales * Guaranteed fares and 24-hour hold policy for American Airlines flights (including American Airlines flights operated by codeshare partners) * Handling of customer issues * Lowest fare availability * Service with domestic codeshare partners * Ticket refunds * Other travel policies Accommodation of customers with special needs This content can be expanded -- accommodation","label": "Updated January 29, 2024","level": "2","name": "Updated January 29, 2024""content": "American Airlines and American Eagle's unaccompanied minor service allows children in certain circumstances to travel alone on our flights. Guidelines for our unaccompanied minor service: * Children 5 years and older may travel on AmericanAirlines or American Eagle on non-stop or same-plane flights. * Children must be 8-14 years old to travel itineraries requiring a connection on American Airlines or American Eagle, connecting flights must be through the following airports: CLT, DCA, DFW, JFK, LAX, LGA, MIA, ORD, PHL, PHX. * Children 15-17 years old are considered young adults and are not required to use the unaccompanied minor service. However, the unaccompanied minor service is available to them upon request and under the same provisions.* American Airlines and American Eagle do not accept unaccompanied children (UMNR) when their itinerary includes a connection to / from another airline, including codeshare and **one**world\u00ae partners. * A charge is assessed for providing unaccompanied minor service. When taking your child's reservation, we will request telephone numbers for parents or guardians. At check-in, you will also be asked to provide names, addresses, and home and business phone numbers for you and the party meeting your child. This information is used in identifying the party meeting your child upon arrival, and it will assist us in contacting you if necessary. We recommend checking in early at the airport to ensure that you have adequate time to complete the required paperwork and pre-board your child. The welfare of unaccompanied minors is a matter we take very seriously. In advance or upon check-in, if there is a possibility that your child's flight might be delayed or misconnect for any reason, we may change your child's flight schedule, including the departure date. An airport representative will assist your child at the connecting city, escort your child to his or her next flight, and deliver your child to the person meeting him or her at the destination airport.We do not accept reservations for an unaccompanied child if it would mean his or her connecting flight would be the last flight of the day.Additional details on our unaccompanied minor service are available from Reservations. Helpful suggestions: * You should stay with your child until he or she boards the aircraft and remain at the airport until the flight leaves the gate * Provide your child with an extra copy of his or her contact phone numbers * Before going to the airport, explain what will happen during the child's trip and let them know what to expect; this may include teaching him or her about airplanes, sounds, connecting flights, etc. * Provide your child with books, food or other items which can help entertain him or her during the trip * Let us know if it is your child's first flight so that, if time permits, we can give him or her a little extra attention","label": "Unaccompanied minor service","level": "2","name": "Unaccompanied minor service"},{"content": "American Airlines and American Eagle endeavor to provide passengers with disabilities dignified, professional, and courteous service at all times. We have a team of employees who regularly consult with disability advisory groups on how we can improve airline accessibility and the quality of our service.","label": "Customers with disabilities","level": "2","name": "Customers with disabilities"{"content": "American Airlines and American Eagle Reservations Offices maintain a staff of Special Assistance Coordinators (SACs) whose function is to make pretravel arrangements for our customers with disabilities. They have been specifically trained to work with customers who have self-identified as having a disability and are requesting special assistance. The Reservations representative who responds to the customer's initial call passes pertinent information to a SAC. They will then personally contact the customer to arrange their special needs and provide a toll-free number for followup questions. This information is communicated to Airport and Flight Services personnel to ensure all American personnel are notified and prepared to fulfill the customer's request.","label": "Reservations special assistance coordinators","level": "2","name" : "Reservations special assistance coordinators"},{"content": "American Airlines and American Eagle block a limited number of seats on each aircraft to accommodate customers who identify themselves as having a qualified disability. Adjacent seats are provided, under certain circumstances, for customers with disabilities who must travel with a companion for assistance ","label": "Pre-reserved seats","level": "2","name": "Pre-reserved seats"{’"content": "Canes, crutches and walkers capable of being collapsed small enough to fit into approved overhead and under seat stowage areas do not count toward your carry-on baggage item limit. They must be small enough to be stowed in such a manner as not to protrude into any seating row floor space or main aisle. Items such as seat cushions, detachable control boxes, armrests or footrests may also be carried on board with you.Electronic medical assistive devices such as CPAP machines, respirators, ventilators that use non-spillable batteries may be accepted in the cabin on an individual basis provided they meet certain safety testing standards ","label": "Carry-on assistive devices","level": "2","name": "Carry-on assistive devices"},{"content": "American Airlines and American Eagle accept motorized and nonmotorized assistive devices fortransport. When necessary, we will disassemble and reassemble wheelchairs or assistive devices for customers when they travel. American Airlines and American Eagle provide storage for one passenger's collapsible, manual wheelchair in the cabin of each aircraft. This service is available on a first-come, first-served basis and has priority over canyon baggage belonging to other customers who board at the same city, provided the customer follows the pre-boarding procedure. In-cabin stowage space for assistive devices cannot be pre-reserved, but American Airlines and American Eagleaccept canyon wheelchairs, provided they collapse to fit in an overhead bin or under a seat. Non-collapsible wheelchairs / scooters are accepted as checked baggage. These items can be checked at the main ticket counter or the departure gate. American Airlines and American Eagle offer a special wheelchair service that allows you to check your wheelchair at the departure gate and claim it at your arrival gate free of charge. Although you can arrange to claim your chair at the connecting city, we recommend that the chair be checked through to the final destination. The additional time required to claim and recheck your chair at the connecting city may compromise your ability to make your connection.American Airlines and American Eagle have wheelchairs and our representatives provide wheelchair service at connecting airports. Please let us know you will need this assistance when making your reservation.Assistive devices approved for cabin transport do not count toward the combined number of checked and canyon baggage a passenger is allowed free of charge, nor do they count toward the limit on canyon items you may bring on board.","label": "Wheelchairs","level": "2","name": "Wheelchairs""content": "If you have special needs, upon request, pre-boarding assistance will be provided to you, allowing you the opportunity to be seated prior to general boarding. A special aisle chair is available to assist you if you are unable to walk, and all of our jet aircraft are equipped with specially designed seats that feature moveable aisle armrests to help make seating easier.","label": "Boarding assistance","level": "2","name": "Boarding assistance"},{"content": "Customers requiring inflight medical oxygen may use only FAA-approved portable oxygen concentrators (POCs) during all phases of the flight. A 48-hour advance notice is required to complete medical paperwork and to ensure that an adequate supply of fully-charged batteries is available to power the device. Please contact Reservations at 1-800-433-7300 to arrange for assistance. FAA approved POC: * AirSep LifeStyle - RTCA sticker required * AirSep FreeStyle * AirSep Free Style 5 * AirSep FOCUS * (Caire) SeQual eQuinox / Oxywell - model 4000\\*\\* * Devilbiss iGO * Inogen One * Inogen G2 * Inogen One G3 * Inova Labs - Lifechoice - (model number XYC100) * Inova Labs LifeChoice Activox * Invacare XPO2 includes model numbers (XPOIOO and XPOIOOB) * Invacare SOLO2 * Oxlife Independence or 02 Concepts Oxlife Independence * Oxus, Inc. RS-00400 POC * Precision Medical EasyPulse * Respironics Inc's EverGO * Respironics SimplyGo * SeQual Eclipse models 1, 2 and 3\\* * SeQual Eclipse 3 or 5 -1000BW* * SeQual SAROSW* * Vbox Trooper \\* SeQual Eclipse (models 1000, 1000A, and 1000B), and SeQual Eclipse SAROS may not be used on Republic Airways. These models are too large to fit under the seat for use on board the aircraft. These models may be transported in the cabin as assistive devices and stowed in the overhead compartments, but these models cannot be used on board the aircraft.\\*\\* Approved for use on American mainline aircraft; however, only approved on a case-by-case basis on Envoy aircraft.","label": "Medical oxygen","level": "2","name": "Medical oxygen""content": "We have employees at airports who are trained as Complaint Resolution Officials (CROs) and are available during operating hours. A Corporate Complaint Resolution Official is also available to assist CROs 24 hours per day, seven days per week. Baggage delivery This content can be expanded -- baggagedelivery American Airlines and American Eagle have devoted extensive resources and efforts to ensure our customers and their bags travel on the same flight. In the event your checked bags do not arrive on your flight, reasonable efforts will be made to ensure that the bag is returned to you within 24 hours for flights within the U.S. Since we cannot guarantee the delivery of baggage carried by another airline, our goal to return bags within 24 hours applies only when we are the carrier taking you to your final destination. When checked baggage is delayed, an American Airlines / American Eagle Baggage Service Representative will compensate reasonable interim expenses to purchase items necessary for immediate use. Returning your bags may take longer on international flights due to flight duration, frequency of flights, or Customs & Immigration procedures at the destination airport. American Airlines and American Eagle attempt to contact customers whose unclaimed checked baggage contains contact information. If American determines that your baggage is lost, you will be reimbursed for any checked baggage charges. If you are unable to locate a bag upon arrival, you should notify a baggage service agent prior to leaving the airport. At that time, the bag delivery procedure will be explained, and you will be provided with a delayed baggage receipt explaining the baggage recovery process. You can also check the status of bags online. Delayed bag search - Opens another site in a new window that may not meet accessibility guidelines, -https: / / app.nettracer.aero / pax / american / auth / login Once a delayed or misrouted bag is located, a delivery company will return the bag to your local address at our expense. There are some circumstances that may inhibit our ability to return your bags within 24 hours: * No local name / address / phone numbers are provided * You are located at a remote location or an \"unreachable\" address, such as a cruise ship or a camping site * You changed your delivery address, but did not notify us * We have limited flight schedules to your destination * Operational circumstances prevent American Airlines or American Eagle from being able to locate or deliver your bags within this timeframe A record of your delayed baggage report is available and includes the contact information and delivery address you provided when you filed your report. Visit our delayed bags page for more information. Delayed bags - / il8n / travel-info / baggage / delayed-or-damaged-baggage.jsp","label": "Complaint resolution officials for customers with special needs", "level": "2","name": "Complaint resolution officials for customers with special needs" },"content": "Although statistics suggest it is highly likely your luggage will arrive with you, there is a possibility it might inadvertently be misrouted. For this reason, it is important to place your name, address, and telephone number(s) on the outside and inside of your bag. This will also make your bag easier to identify when claiming it at your destination. You should also carry certain items onboard with you. These could include, but are not limited to, travel documents, medicationjewelry, cash, keys, business documents, electronics, or other items of particularly high personal value. Baggageliability This content can be expanded -- baggageliability Domestic travel is defined as travel in which the itinerary remains within the U.S., including Puerto Rico and / or the U.S. Virgin Islands. American Airlines and American Eagle will reimburse up to $3,800 per ticketed domestic customer for lost or damaged checked baggage, subject to claim verification. The specific procedures for making lost or damaged baggage claims will be provided to you at the time a claim is filed. We will not pay for normal wear and tear on checked baggage or for loss or damage to belongings you carry on board in the passenger cabin. Certain items (e.g., cashjewelry, cameras) are excluded from liability for lost or damaged checked baggage. A complete list of the limitations and exclusions is in our conditions of carriage and is also available upon request. International travel is defined as travel in which the itinerary contains one or more segments of travel outside of the U.S., including Puerto Rico and / or the U.S. Virgin Islands. For travel governed by the Warsaw Convention, we will reimburse up to $9.07 per pound, up to a maximum weight of 70 pounds per checked bag (a maximum of $634.90 per checked bag). For travel governed by the Montreal Convention, we will reimburse up to 1,288 SDR's per ticketed customer. These limits may be lower for additional pieces to certain international destinations or when using connecting carriers. Federal regulations require that checked baggage remain unlocked. Improperly packed items (e.g., boxes) or those considered fragile (e.g., paintings) may not be covered under American Airlines or American Eagle's baggage liability provisions. If the value of the contents of a checked bag exceeds our standard coverage you may purchase additional baggage insurance. This insurance, known as excess value coverage, may be purchased at the ticket counter when checking in. Excess value insurance does not cover damage. Not all high value items are covered by this insurance, so you should verify with the agent that your specific belongings are included. Check-in requirements and baggage acceptance cutoff times This content can be expanded - checkin Make sure you have enough time to check in, check your baggage and get through security: *WCheck-in and arrival - / il8n / travel-info / check-in-and-arrival.jsp *WMinimum time for checking in and arriving at the gate — / il8n / travel-info / arrival-times.jsp * WCutoff times for checking bags - / il8n / travel-info / arrival-times.jsp You can also check the latest travel alerts to learn about circumstances that could affect schedules, airport operation and parking, as well as flight status information for gates and times. * WTravel alerts - / i 18n / travel-info / travel-alerts.j sp * WGet flight status information - / travellnformation / flights / status Customer loyalty program - AAdvantage\u00ae This content can be expanded -customerloyalty The AAdvantage\u00ae program, the first frequent traveler program in the industry, began in 1981. Anyone wishing to enroll in the AAdvantage\u00ae program can do so instantly at any American Airlines or American Eagle ticket counter, online at the aa.com website, or by calling 800-433-7300, AAdvantage\u00ae members earn mileage credits by flying on American Airlines, American Eagle and **one**world\u00ae carriers, as well as on other airline participants. Members also earn mileage credits when using affiliated credit / debit cards or by purchasing products / services from participating car rental and hotel companies, retailers and a wide variety of service providers. In fact, over half of all miles earned are for products and services other than flying. Members can earn miles at more than 1,000participating companies, which include over 30 hotel chains representing more than 60 brands, more than 20 airlines, eight car rental companies,12 financial companies and over 350 brand name retailers. In addition, members can earn miles when making purchases with one of more than 100 affinity card products in over 40 countries. AAdvantage\u00ae terms and conditions - / il8n / aadvantage-program / aadvantage-terms-and-conditions.jsp Your AAdvantage\u00ae miles do not expire if you have at least one qualifying transaction within the previous 24 months. All types of mileage-earning and redeeming transactions are included - anything from a mileage-earning flight on American Airlines or American Eagle to purchases using the Citi\u00ae / AAdvantage\u00ae card. If you\u2019re a primary AAdvantage\u00ae credit cardholder or under 21 years of age, you are not subject to the 24-month qualifying activity period. AAdvantage\u00ae benefits are numerous and varied. You can claim a flight award, for any seat on any flight, with no blackout dates. All awards can be issued in any name you designate, subject to the rules prohibiting the sale or barter of awards. In addition, you can donate your miles to the Miles for Kids in Need\u00ae program, which provides transportation for seriously ill children. American Airlines reserves the right to change AAdvantage\u00ae program rules, regulations, Flight awards and special offers at any time without notice and to end the AAdvantage\u00ae program with six months notice. Any such changes may affect your ability to use the mileage awards or credits that you have accumulated. Members may not be able to obtain all offered awards at all times or use awards for all destinations or on all flights. Unless specified, AAdvantage\u00ae miles earned through promotions / offers do not count toward status qualification or Million MilerSM status. Flight awards, mileage accrual and special offers are subject to government regulations. American Airlines is not responsible for products or services offered by other participating companies. Delays, cancellations and diversion events This content can be expanded - delays","label": "Helpful suggestions:","level": "2","name": "Helpful suggestions:"},{"children": [{"content": "We will provide timely and frequent updates about known delays, cancellations and diversions within 30 minutes of becoming aware that the flight is impacted. We will strive to provide the best available information about the duration of delays and the flight's anticipated departure time. * We will contact you in advance, whenever reasonable, at the phone numbers you provided in your reservation. * We will provide phone, email and text updates for customers who sign up for flight status notifications on aa.com. * If you booked with a travel agency, a team member will try to contact the agency if we don\u2019t have your phone number. * If your flight is diverted due to a medical emergency, severe weather, or other situation, we will notify you at the gate before departure, when possible, or make an announcement on board.","label": "How we notify you","level": "3","name" : "How we notify you"],}"content": "We expect, as you do, that American Airlines will operate flights on time. On occasion, the weather or other challenges can make it difficult, if not impossible, to stay on schedule. When a delay or cancellation does occur, our goal is to get you to your destination safely, and as quickly as possible. Whenever possible give us a cell phone number when you make a reservation or check in and sign up for flight status notifications on aa.com.","label": "Assistance when your flight has been delayed or canceled", "level": "2","name": "Assistance when your flight has been delayed or canceled""children": ["content": "When your flight is canceled, significantly delayed, or a delay could cause you to miss your connection, we\u201911 rebook you on our next flight with available seats at no additional cost. If no American Airlines flights are available until the next day, we\u201911 rebook you on one of our partner airlines at no additional cost. In cases where a delay or cancellation is caused by us, upon request we\u201911 give you: * A voucher for an approved hotel with available rooms if you are delayed overnight if you\u2019re away from your city of residence. * Transportation to a hotel and back to the airport by hotel shuttle I third-party transportation service, or a transportation voucher. * Meal vouchers if your delay is 3 or more hours after your scheduled departure. If we can\u2019t provide you with a voucher for an approved hotel, we\u201911 reimburse you for reasonable hotel costs. If a hotel shuttle / third-party transportation service isn\u2019t available, or we can\u2019t provide you with a transportation voucher, we\u201911 reimburse you for reasonable transportation costs.","label": "Delays and cancellations caused by us","level": "3","name": "Delays and cancellations caused by us"},{"content": "If your flight is diverted to another city and we don\u2019t board to your final destination before 11 :59 p.m., we\u201911 give you: * A voucher for an approved hotel with available rooms. * Transportation to a hotel and back to the airport by hotel shuttle / third-party transportation service, or a transportation voucher. * Meal vouchers if you are delayed in the diversion city for more than 3 hours. If we can\u2019t provide you with a voucher for an approved hotel, we\u201911 reimburse you for reasonable hotel costs. If a hotel shuttle / third-party transportation service isn\u2019t available, or we can\u2019t provide you with a transportation voucher, we\u201911 reimburse you for reasonable transportation costs.","label": "Diversions","level": "3","name" : "Diversions"},"content": "When your flight is canceled, significantly delayed, or a delay could cause you to miss your connection, we\u201911 rebook you on our next flight with available seats at no additional cost. If the delay or cancellation is caused by events beyond our control (like weather), you are responsible for your own overnight accommodations, meals and incidental expenses. American Airlines agents may be able to help you find a hotel. If you decide not to fly because of a flight cancellation or a significant delay, you can request a refund for the remaining ticket value and related optional fees. Request a refund - Opens another site in a new window that may not meet accessibility guidelines. - https: / / refunds.aa.com / refunds","label": "Delays beyond our control (like weather)","level": "3","name": "Delays beyond our control (like weather)"}],"content":"label": "Flight delays and cancellations","level": "2","name": "Flight delays and cancellations""content": "Our top priority is the safety and well-being of our customers, and we are focused on having all our flights arrive and deplane on schedule. On very rare occasions, there may be extraordinary events that result in lengthy onboard delays on the ground (other than on an active runway or taxiway) with no access to a terminal gate for more than 2 hours. We have detailed contingency plans at every U.S. airport to address these situations. In such events, we will make every reasonable effort to ensure your essential needs, including: * Snacks * Water * Restroom access * Basic medical assistance Every American Airlines U.S. airport team has an operational contingency plan to address these needs, including coordination with the airport and other airlines. Each plan designates a local control person to coordinate and communicate with our operations team at headquarters. Family seating This content can be expanded - familyseating Our goal is to have families seated together. We guarantee children 14 and under will be seated adjacent to an accompanying adult at no additional cost, including Basic Economy fares, if you meet all these conditions: * Everyone is booked in the same reservation * Adjacent seats are available in the same class of service your child is ticketed at the time of booking * You choose seats for the entire reservation or skip seats for the entire reservation (if you choose Basic Economy, then skip choosing seats for the entire reservation) * You don\u2019t make a change to your seat assignments once they\u2019re assigned to you * The plane\u2019s seat layout allows it based on the number of children in your reservation * The original flight isn\u2019t switched to a smaller aircraft If these conditions aren\u2019t met, we will still try to seat children adjacent to an accompanying adult. Flights with oversales This content can be expanded - flightwithoversales American Airlines and American Eagle, like mostairlines, overbook flights. We do this because historical information shows that some people do not cancel their reservations when they change their travel plans. Overbooking is done in the best interest of both customers and the airline. Without the revenue produced by filling seats that would otherwise go empty, every airline would have to compensate by raising fares. And just as important, selling our seats in this manner allows us to say \"yes\" rather than \"no\" a lot more often when customers call for a seat on the flight of their choice. Just because a flight is overbooked, it does not necessarily mean that customers with reservations will not be accommodated on their flight. The overwhelming majority of overbooked flights depart with all customers who have reservations and are present for departure. If at departure time more customers with confirmed reservations are present than there are seats available, gate agents will first ask for volunteers who are willing to give up their seats in exchange for compensation and a confirmed seat on a later flight. On extremely rare occasions, a customer may be denied boarding on an involuntary basis, if a sufficient number of volunteers are not obtained. In such events, we will usually deny boarding based upon check-in time, but we may also consider factors such as severe hardships, fare paid, and status within the AAdvantage\uOOae program.With few exceptions, persons denied boarding involuntarily are entitled to compensation under federal law. Additional information concerning our overbooking policies can be found in our conditions of carriage. Upon request, reservations representatives or airport agents will advise you if your flight is overbooked at the time your reservation is made or during airport check-in. Keep in mind that most overbooked flights do in fact have sufficient seats to accommodate all customers who are present for departure. Lowest fare availability This content can be expanded -lowestfareavailability Customers calling our reservations office or visiting our airline ticket offices or ticket counters will be offered the lowest available fare, exclusive of Internet only fares, when specific dates and times are provided. If you do not provide specific information about your travel requirements, we will provide you with a range of fares for your general needs. In the event the lowest available fare is not quoted, American's liability is limited to the difference between the fare quoted and the lowest available fare for which the customer was eligible at that time. If lower AA fares are available online or elsewhere, we will advise this is the case. Helpful suggestions: * Basic Economy is our lowest fare available but comes with restrictions * State preferred dates of travel * Bear in mind that most of the lowest fares usually require advance purchase, minimum stay and flexible travel plans * Make reservations and purchase tickets as far in advance as possible * Avoid peak travel days immediately prior to or after a holiday * Consider near-by alternate airports * Request specific fares, if eligible (e.g., senior citizen fares or active military fares)* Check fare options on our website, aa.com In the U.S., please contact American Airlines or American Eagle at 800-433-7300 for further information. If assistance is needed in Spanish, you may dial 800-633-3711. If you have a hearing or speech impairment, you may contact American Airlines or American Eagle at 800-543-1586.International reservations are handled through local offices, and those numbers can be found in local telephone directories. Refunds This content can be expanded - refunds American Airlines strives to process eligible refunds after we receive all required information. Keep in mind, some optional products and services tickets are not refundable. Read our refunds FAQs for details. Refunds FAQs - / il8n / customer- service / faqs / customer-service-faqs.jsp#refunds Some countries have different rules about refunds, and if so, we will follow those rules.","label": "Essential customer needs during extraordinarydelays",level": "2",name": "Essential customer needs during extraordinary delays"children": ["content": "For all eligible tickets and optional products and services bought within the LT.S. with a credit card, refunds will be processed within 7 business days of receiving the request and required information. Keep in mind, it may take up to 2 billing cycles to show on your credit card statement. Refunds for credit card purchases will be made only to the credit card account. Service charges are collected for some refunds.", "label": "Refunds to a credit card","level": "3","name": "Refunds to a credit card"},{"content": "If you paid for tickets and optional products and services with cash or a check, we\u201911 process the refund within 20 business days of receiving the request and required information.","label": "Cash or check refunds","level": "3","name": "Cash or check refunds""content":"label": "Tickets, optional products and services","level": "2","name": "Tickets, optional products and services"},"content": "You can request a refund of eligible taxes included in the ticket price. If eligible, we'll refund the original form of payment. * Destination-specific taxes not imposed by us may be eligible * Taxes we must pay whether you travel are not eligible ", "label": "Refunds of taxes","level": "2","name": "Refunds of taxes"},{"children": [{"content": "American Airlines Attention: Passenger Refunds 4000 E. Sky Harbor Blvd. Phoenix, AZ 85034","label": "Mail to:","level": "3","name": "Mail to:""content": "Submit all required documents and informationwith your refund request including: * The receipt for a valid ticket or optional products and services before the expiration date. Keep in mind, tickets and optional services expire 1 year from the date they are issued. * The ticket number for electronic tickets * A brief written explanation * Your name, address, and phone number Please note, some requests may take longer, for example, tickets bought outside the U.S. in another currency or tickets that require research or verification. We are not liable for longer processing times. 24-hour refund This content can be expanded - guaranteedfares When you buy a ticket on aa.com or through American Airlines Reservations at least 2 days prior to departure, you have up to 24 hours from the time you first buy your ticket to request a refund. The 24 hour refund policy applies to all ticket types, but you have to cancel your trip to get a refund. \\* Reservations booked as part of group block do not qualify for the 24 hour refund.AAdvantage\u00ae award tickets can be canceled anytime on aa.com. We\u201911 reinstate your miles and refund eligible taxes and fees up to 1 year after the ticket issue date, but you have to cancel your trip before your first flight departs. There is no fee to reinstate your miles. Canceling flight and upgrade awards and reinstating miles - / i 18n / aadvantage-program / miles / redeem / award-travel / award-travel.j sp#cancelingandreinstating Refunds will be issued in the full amounts, less: * Sale-imposed nonrefundable taxes * Service charge for booking through Reservations * Taxes we must pay whether or not you travel Some countries have different rules about refunds, and if so, we will follow those rules. Our normal refund policies apply after 24 hours. If you make changes to your trip after 24 hours you may owe any difference in ticket price and change fees.","label": "Required documents and information","level": "3","name": "Required documents and information"},{"content": "To cancel your trip, find your trip on aa.com or contact Reservations and refunds will be processed automatically. Find your trip - reservation / view / find-your-reservation","label": "Cancel your trip","level": "3","name": "Cancel your trip"},{"content": "Ifyou can\u2019t cancel your trip online orneed additional information call Reservations for help. WContact Reservations - / i 18n / customer- service / contact-american / reservations-and-ticket-changes.j sp \\*If you bought your ticket through a travel agency or another booking source, contact them for a refund. Other travel policies This content can be expanded - othertravelpolicies","label": "Call Reservations","level": "3","name" : "Call Reservations"}],"content": "You can request a refund online, by contacting Reservations, or by mail or fax. * Request a refund online - Opens another site in a new window that may not meet accessibility guidelines. - https: / / refunds.aa.com / refunds * Contact Reservations - / il8n / customer-service / contact-american / reservations-and-ticket-changes.jsp","label": "Request a refund","level": "2","name": "Request a refund""content": "American Airlines and American Eagle Reservations representatives will advise you when it will be necessary for you to move from one aircraft to another during your travel, including when the flight number remains the same.","label": "Aircraft assignment","level": "2","name": "Aircraft assignment"{’"content": "American Airlines and American Eagle tickets are priced and sold based on the itinerary, and not as individual flight segments. If you foresee or must make a change to the itinerary while en route, you must contact American Airlines or American Eagle to determine how this may affect the ticket and remaining travel plans. For example, if you do not show up for one segment of your itinerary, we may cancel your reservations on all remaining flight segments.","label": "Travel itinerary policy","level": "2","name": "Travel itinerary policy""content" : "American Airlines and American Eagle Reservations representatives are able to provide you with a wide variety of information related to the American Airlines and American Eagle fleets. Aircraft configuration, seat size, and seat pitch are just a few examples of information that can be obtained from our Reservations representatives. This information is also available at the aa.com website.","label": "Aircraft configuration","level": "2","name": "Aircraft configuration"{’"content": "American Airlines and American Eagle commit that we will advise you of any changes to your itinerary in a timely manner and will contact you via contact information which you provided in your reservations for any changes to flight schedules after booking.","label": "Itinerary changes","level": "2","name": "Itinerary changes""content": "American Airlines and American Eagle will make their Customer Service Plan readily available to all of our domestic Codeshare partners and, to the extent possible, make every reasonable effort to ensure our partners offer comparable levels of service","label": "Service with domestic codeshare partners","level": "2","name": "Service with domestic codeshare partners"},"content": "Our goal is to be a service and product leader in the airline industry. When customer service issues arise, we try to resolve them at the first point of contact. As a result, we have devoted resources to the front line and rely on our personnel to solve most issues throughout the travel journey. After travel, our Customer Relations department is also dedicated to addressing customer comments, compliments and concerns. While we typically respond within a few days; customers can expect at least an acknowledgement within 30 days and resolution within 60 days. All customer feedback is personally read and a response is sent to the customer. Customer Relations information, along with other customer insight from things like focus groups, surveys or media, is assessed and shared throughout American Airlines and American Eagle. We have the privilege and responsibility to address and initiate appropriate action to improve our customer experience \u2013 we view our customer\u2019s voice as the most valuable tool to enhance product and services. Contact Customer Relations - / i 18n / customer-service / contact-american / customer-relations.jsp Should you prefer to use U.S. Mail, the address is: U.S. Mail / Overnight Mail American Airlines Customer Relations 4000 E. Sky Harbor Blvd. Phoenix, AZ 85034 Helpful suggestions: * Be as specific as possible, including dates and flight numbers * Provide all supportive documentation, such as copies ofyour tickets and certificates Passenger rights \u2013 Canada - / il8n / customer-service / support / canada-passengers.jsp?locale=en_CA Back to top -- breadcrumb","label": "Handling of customer issues","level": "2","name": "Handling of customer issues"}],"content":"label": "Customer service plan","level": "1","name": "Customer service plan"],}"content":"label": "Document top header","level": "0","name": "Document top header"1

Claims

What is claimed is:

1. A method, comprising:retrieving, by a computer system, a plurality of pieces of content that embody the one or more principles for a specific domain;generating, by the computer system, one or more prompts from each of the plurality of pieces of content, the one or more prompts being data to align an artificial intelligence model to the one or more principles for the specific domain;wherein generating the one or more prompts further comprises creating, by the computer system for each piece of retrieved content, a structured representation of the pieces of data in each piece of retrieved content; performing, by the computer system on the structured representation for each piece of content, a recursive summarization to generate a summary that is stored back into the structured representation for each piece of content; generating, by the computer system based on the structured representation for each piece of content that includes the summary, at least one question and answer pair (QA pair) that further comprises performing, by the computer system based on the structured representation for each piece of content that includes the summary, a relevance assessment of each node in the structured representation of the pieces of data in each piece of retrieved content to identify a node relevant to a fine tuning direction for the artificial intelligence model: generating, by the computer system, an initial QA pair for each relevant node in the structured representation of the pieces of data in each piece of retrieved content, performing, by the computer system on each initial QA pair, an answer refinement to generate a refined QA pair; and adding, by the computer system, the refined QA pair to the summary of the node; and generating, by the computer system using the QA pair for each piece of content, the one or more prompts to align the artificial intelligence model to the one or more principles for the specific domain; training, by a second computer system, using the one or more prompts, the artificial intelligence model to align to the one or more principles for the specific domain; andgenerating, by the second computer system in response to a user query using the aligned artificial intelligence model, a response to the user query that is aligned to the one or more principles for the specific domain.

2. The method of claim 1, wherein the structured representation of each piece of retrieved content is a document tree.

3. The method of claim 2, wherein the document tree further comprises a plurality of nodes wherein each node stores one of a piece of text and a structural element of the piece of retrieved content.

4. The method of claim 1, wherein the artificial intelligence model is one of a large language model and a large multimodal model.

5. The method of claim 1, wherein the specific domain is one of an industry standard, a civility score, an enterprise domain, a set of pieces of content from a computer and a blog post.

6. The method of claim 1 further comprising generating, by the second computer system using the one or more prompts, one or more alignment processes, generating a response to a user query in response to a trained artificial intelligence model and aligning the generated response from the trained artificial intelligence model based on the one or more alignment processes to generate an aligned response to the user query that aligns to the one or more principles for the specific domain.

7. The method of claim 6, wherein the artificial intelligence model is one of a large language model and a large multimodal model.

8. The method of claim 6, wherein the trained artificial intelligence model is one of a large language model and a large multimodal model.

9. A system comprising:a teacher computer system having a processor and a plurality of instructions executed by the processor;a student computer system having a processor and a plurality of instructions executed by the processor;the processor of the teacher computer system executing the plurality of instructions to be configured to:retrieve a plurality of pieces of content that embody one or more principles for a specific domain; create, for each piece of retrieved content, a structured representation of the pieces of data in each piece of retrieved content;perform, on the structured representation for each piece of content, a recursive summarization to generate a summary that is stored back into the structured representation for each piece of content;generate, based on the structured representation for each piece of content that includes the summary, at least one question and answer pair (QA pair) that further perform, based on the structured representation for each piece of content that includes the summary, a relevance assessment of each node in the structured representation of the pieces of data in each piece of retrieved content to identify a node relevant to a fine tuning direction for the artificial intelligence model, generate, an initial QA pair for each relevant node in the structured representation of the pieces of data in each piece of retrieved content, perform, on each initial QA pair, an answer refinement to generate a refined QA pair and adding the refined QA pair to the summary of the node of the structured representation that is used to generate the one or more prompts; generate, using the QA pair for each piece of content, one or more prompts from each of the plurality of pieces of content, the one or more prompts being data to align an artificial intelligence model to the one or more principles for the specific domain; and communicate the one or more prompts to the student computer system; andwherein the processor of the student computer system is configured to generate, using the one or more prompts, one or more alignment processes, generate a response to a user query in response to a trained artificial intelligence model and align the generated response based on the one or more alignment processes to generate an aligned response that aligns to the one or more principles for the specific domain.

10. The system of claim 9, wherein the structured representation of each piece of retrieved content is a document tree.

11. The system of claim 10, wherein the document tree further comprises a plurality of nodes wherein each node stores one of a piece of text and a structural element of the piece of retrieved content.

12. The system of claim 9, wherein the artificial intelligence model is one of a large language model and a large multimodal model.

13. The system of claim 9, wherein the specific domain is one of an industry standard, a civility score, an enterprise domain, a set of pieces of content from a computer and a blog post.

14. The system of claim 9, wherein the processor of the teacher computer system is further configured to communicate the one or more prompts to the student computer system and wherein the processor of the student computer system is configured to train, using the one or more prompts, an artificial intelligence model to align to the one or more principles for the specific domain and generate, in response to a user query using the aligned artificial intelligence model, a response to the user query that is aligned to the one or more principles for the specific domain.

15. The system of claim 14, wherein the artificial intelligence model is one of a large language model and a large multimodal model.

16. The system of claim 9, wherein the trained artificial intelligence model is one of a large language model and a large multimodal model.

17. An apparatus, comprising:a computer having a processor wherein the computer receives a query and has an aligned artificial intelligence model (AIM), wherein the aligned AIM generates an aligned response to the query that is aligned to one or more principles of a specific domain;the aligned AIM created by the processor executing instructions and the processor being configured to:retrieve a plurality of pieces of content that embody the one or more principles for a specific domain;generate, a first plurality of question answer pairs (QA pairs) aligned to the specific domain;refine the first plurality of QA pairs to generate an acceptable plurality of QA pairs aligned to the specific domain; andtrain an untrained AIM using the acceptable plurality of QA pairs to produce the trained AIM that generates the aligned response to the query.

18. The apparatus of claim 17, wherein the processor that generates the first plurality of QA pairs is further configured to create, for each piece of retrieved content, a structuredrepresentation of the pieces of data in each piece of retrieved content, perform, on the structured representation for each piece of content, a recursive summarization to generate a summary that are stored back into the structured representation for each piece of content, generate, based on the structured representation for each piece of content that includes the summary, at least one question and answer pair (QA pair) and generate, using the QA pair for each piece of content, one or more prompts from each of the plurality of pieces of content, the one or more prompts being data to align an artificial intelligence model to the one or more principles for the specific domain.

19. The apparatus of claim 18, wherein the processor configured to refine the first plurality of QA pairs is further configured to perform, based on the structured representation for each piece of content that includes the summary, a relevance assessment of each node in the structured representation of the pieces of data in each piece of retrieved content to identify a node relevant to a fine tuning direction for the artificial intelligence model, generate, an initial QA pair for each relevant node in the structured representation of the pieces of data in each piece of retrieved content, perform, on each initial QA pair, an answer refinement to generate a refined QA pair and adding the refined QA pair to the summary of the node of the structured representation that is used to generate the one or more prompts.

20. The apparatus of claim 19, wherein the structured representation of each piece of retrieved content is a document tree.

21. The apparatus of claim 20, wherein the document tree further comprises a plurality of nodes wherein each node stores one of a piece of text and a structural element of the piece of retrieved content.

22. The apparatus of claim 17, wherein the untrained artificial intelligence model is one of a large language model and a large multimodal model.

23. The apparatus of claim 17, wherein the specific domain is one of an industry standard, a civility score, an enterprise domain, a set of pieces of content from a computer and a blog post.

24. A method, comprising:retrieving, by a computer system, a plurality of pieces of content that embody the one or more principles for a specific domain;creating, by the computer system for each piece of retrieved content, a structured representation of the pieces of data in each piece of retrieved content;performing, by the computer system on the structured representation for each piece of content, a recursive summarization to generate a summary that are stored back into the structured representation for each piece of content;generating, by the computer system based on the structured representation for each piece of content that includes the summary, at least one question and answer pair (QA pair);generating, by the computer system using the QA pair for each piece of content, the one or more prompts to align the artificial intelligence model to the one or more principles for the specific domain; andtraining, by the computer system, using the one or more prompts, an artificial intelligence model to align to the one or more principles for the specific domain.

25. The method of claim 24 further comprising generating, by a second computer system in response to a user query using the aligned artificial intelligence model, a response to the user query that is aligned to the one or more principles for the specific domain.

26. The method of claim 25, wherein generating the at least one QA pair further comprises performing, by the computer system based on the structured representation for each piece of content that includes the summary, a relevance assessment of each node in the structured representation of the pieces of data in each piece of retrieved content to identify a node relevant to a fine tuning direction for the artificial intelligence model; generating, by the computer system, an initial QA pair for each relevant node in the structured representation of the pieces of data in each piece of retrieved content, performing, by the computer system on each initial QA pair, an answer refinement to generate a refined QA pair; and adding, by the computer system, the refined QA pair to the summary of the node.

27. The method of claim 24, wherein the structured representation of each piece of retrieved content is a document tree.

28. The method of claim 27, wherein the document tree further comprises a plurality of nodes wherein each node stores one of a piece of text and a structural element of the piece of retrieved content.

29. The method of claim 24, wherein the artificial intelligence model is one of a large language model and a large multimodal model.

30. The method of claim 24, wherein the specific domain is one of an industry standard, a civility score, an enterprise domain, a set of pieces of content from a computer and a blog post.

31. A system, comprising:a computer system having a processor that executes a plurality of lines of instructions, the processor being configured to:retrieve a plurality of pieces of content that embody the one or more principles for a specific domain;create, for each piece of retrieved content, a structured representation of the pieces of data in each piece of retrieved content;perform, on the structured representation for each piece of content, a recursive summarization to generate a summary that are stored back into the structured representation for each piece of content;generate, based on the structured representation for each piece of content that includes the summary, at least one question and answer pair (QA pair);generate, using the QA pair for each piece of content, the one or more prompts to align the artificial intelligence model to the one or more principles for the specific domain; and train, using the one or more prompts, an artificial intelligence model to align to the one or more principles for the specific domain.

32. The system of claim 31 further comprising a second computer system having a processor that is configured to generate in response to a user query using the aligned artificial intelligence model, a response to the user query that is aligned to the one or more principles for the specific domain.

33. The system of claim 32, wherein the processor configured to generate the at least one QA pair is further configured to perform, based on the structured representation for each piece of content that includes the summary, a relevance assessment of each node in the structured representation of the pieces of data in each piece of retrieved content to identify a node relevant to afine tuning direction for the artificial intelligence model; generate an initial QA pair for each relevant node in the structured representation of the pieces of data in each piece of retrieved content, perform, on each initial QA pair, an answer refinement to generate a refined QA pair and add the refined QA pair to the summary of the node.

34. The system of claim 31, wherein the structured representation of each piece of retrieved content is a document tree.

35. The system of claim 34, wherein the document tree further comprises a plurality of nodes wherein each node stores one of a piece of text and a structural element of the piece of retrieved content.

36. The system of claim 31, wherein the artificial intelligence model is one of a large language model and a large multimodal model.

37. The system of claim 31, wherein the specific domain is one of an industry standard, a civility score, an enterprise domain, a set of pieces of content from a computer and a blog post.

38. A method, compri sing :retrieving, by a computer system, a plurality of pieces of content that embody the one or more principles for a specific domain;creating, on the computer system for each piece of retrieved content, a structured representation of the pieces of data in each piece of retrieved content;performing, on the computer system on the structured representation for each piece of content, a recursive summarization to generate a summary for each piece of content that are stored back into the structured representation for each piece of content;performing, by the computer system based on the structured representation for each piece of content that includes the summary, a relevance assessment of each node in the structured representation of the pieces of data in each piece of retrieved content to identify a node relevant to a fine tuning direction for the artificial intelligence model to generate a first plurality of question answer pairs (QA pairs) aligned to the specific domain;performing, by the computer system on each initial QA pair, an answer refinement to generate a refined QA pair and an acceptable plurality of QA pairs aligned to the specific domain; andadding, by the computer system, the refined QA pair to the summary of the relevant node.

39. The method of claim 38, wherein the structured representation of each piece of retrieved content is a document tree.

40. The method of claim 39, wherein the document tree further comprises a plurality of nodes wherein each node stores one of a piece of text and a structural element of the piece of retrieved content.

41. The method of claim 38 further comprising training an artificial intelligence model (AIM) using the acceptable plurality of QA pairs to align the AIM with the specific domain.

42. The method of claim 41 further comprising receiving a query and generating an aligned response by the aligned AIM to the received query.

43. The method of claim 41, wherein the AIM is one of a large language model and a large multimodal model.

44. The method of claim 43, wherein the specific domain is one of an industry standard, a civility score, an enterprise domain, a set of pieces of content from a computer and a blog post.

45. The method of claim 38 further comprising generating one or more alignment processes using the using the acceptable QA pairs.

46. The method of claim 45 further comprising outputting a response to a query from a trained AIM and adjusting, using the one or more alignment processes, the outputted response from the training AIM to generate an aligned response to the query that is aligned to the specific domain.

47. The method of claim 46, wherein the trained artificial intelligence model is one of a large language model and a large multimodal model.

48. The method of claim 47, wherein the specific domain is one of an industry standard, a civility score, an enterprise domain, a set of pieces of content from a computer and a blog post.

49. A system, comprising:a computer system having a processor and a plurality of lines of instructions executed by the processor so that the computer system is configured to:retrieve a plurality of pieces of content that embody the one or more principles for a specific domain;create, for each piece of retrieved content, a structured representation of the pieces of data in each piece of retrieved content;perform, on the structured representation for each piece of content, a recursive summarization to generate a summary for each piece of content that are stored back into the structured representation for each piece of content;perform, based on the structured representation for each piece of content that includes the summary, a relevance assessment of each node in the structured representation of the pieces of data in each piece of retrieved content to identify a node relevant to a fine tuning direction for the artificial intelligence model to generate a first plurality of question answer pairs (QA pairs) aligned to the specific domain;perform, on each initial QA pair, an answer refinement to generate a refined QA pair and an acceptable plurality of QA pairs aligned to the specific domain; andadd the refined QA pair to the summary of the relevant node.

50. The system of claim 49, wherein the structured representation of each piece of retrieved content is a document tree.

51. The system of claim 50, wherein the document tree further comprises a plurality of nodes wherein each node stores one of a piece of text and a structural element of the piece of retrieved content.

52. The system of claim 49, wherein the computer system is further configured to train an artificial intelligence model (AIM) using the acceptable plurality of QA pairs to align the AIM with the specific domain.

53. The system of claim 52, wherein the computer system is further configured to receive a query and generate an aligned response by the aligned AIM to the received query.

54. The system of claim 52, wherein the AIM is one of a large language model and a large multimodal model.

55. The system of claim 54, wherein the specific domain is one of an industry standard, a civility score, an enterprise domain, a set of pieces of content from a computer and a blog post.

56. The system of claim 49, wherein the computer system is further configured to generate one or more alignment processes using the using the acceptable QA pairs.

57. The system of claim 56, wherein the computer system is further configured to output a response to a query from a trained AIM and adjust, using the one or more alignment processes, the outputted response from the training AIM to generate an aligned response to the query that is aligned to the specific domain.

58. The system of claim 57, wherein the trained artificial intelligence model is one of a large language model and a large multimodal model.

59. The system of claim 58, wherein the specific domain is one of an industry standard, a civility score, an enterprise domain, a set of pieces of content from a computer and a blog post.

60. A method, comprising:retrieving, by a computer system, a plurality of pieces of content that embody the one or more principles for a specific domain;creating, by the computer system for each piece of retrieved content, a structured representation of the pieces of data in each piece of retrieved content;performing, by the computer system on the structured representation for each piece of content, a recursive summarization to generate a summary that are stored back into the structured representation for each piece of content;generating, by the computer system based on the structured representation for each piece of content that includes the summary, at least one question and answer pair (QA pair);generating, by the computer system using the QA pair for each piece of content, the one or more prompts to align the artificial intelligence model to the one or more principles for the specific domain; andgenerating, by the computer system using the one or more prompts, a plurality of alignment processes that adjust the responses from an already trained AIM.

61. The method of claim 60 further comprising generating a response to a user query by an already trained artificial intelligence model and aligning the generated response based on the one or more alignment processes to generate an aligned response to the user query that aligns to the one or more principles for the specific domain.

62. The method of claim 60, wherein the artificial intelligence model is one of a large language model and a large multimodal model.

63. The method of claim 60, wherein the specific domain is one of an industry standard, a civility score, an enterprise domain, a set of pieces of content from a computer and a blog post.

64. The method of claim 60, wherein generating the at least one QA pair further comprises one or more of a retrieval- augmented generation process, an API generation process, a weak labeling generation process and a synthetic instruction generation process.

65. A system, comprising:a computer system having a processor that executes a plurality of lines of instructions, the processor being configured to:retrieve a plurality of pieces of content that embody the one or more principles for a specific domain;create, for each piece of retrieved content, a structured representation of the pieces of data in each piece of retrieved content;perform, on the structured representation for each piece of content, a recursive summarization to generate a summary that are stored back into the structured representation for each piece of content;generate, based on the structured representation for each piece of content that includes the summary, at least one question and answer pair (QA pair);generate, using the QA pair for each piece of content, the one or more prompts to align the artificial intelligence model to the one or more principles for the specific domain; andgenerate, using the one or more prompts, a plurality of alignment processes that adjust the responses from an already trained AIM.

66. The system of claim 65, wherein the processor is further configured to generate a response to a user query by an already trained artificial intelligence model and align the generated response based on the one or more alignment processes to generate an aligned response to the user query that aligns to the one or more principles for the specific domain.

67. The system of claim 65, wherein the artificial intelligence model is one of a large language model and a large multimodal model.

68. The system of claim 65, wherein the specific domain is one of an industry standard, a civility score, an enterprise domain, a set of pieces of content from a computer and a blog post.

69. The system of claim 65, wherein the processor is further configured to generate the at least one QA pair by one or more of a retrieval- augmented generator, an API generator, a weak labeling generator and a synthetic instruction generator.

Images