Sensor-in-the-loop ai agent

Abstract

Embodiments described herein provide for an advanced Al system integrating Large Language Models (LLMs) with healthcare sensor data to enhance personalized user interactions. The software programming executed by a user device obtains sensor data of various categories, such as activity level, stress level, and sleep quality, and generates a sensor aggregation file (or other data format for a user profile summary) with severity metrics, scores, or levels for each category. The system obtains a user-entered LLM prompt and performs a first call to an LLM to generate a refined-LLM prompt based on sensor aggregation file, and then obtains a tailored response to the refined-LLM prompt by performing a second call to the LLM.

Description

076333-1064 / 06906 PATENTSENSOR-IN-THE-LOOP Al AGENTCROSS-REFERENCE TO RELATED APPLICATION

[0001] This non-provisional application claims priority to and the benefit of US Provisional Application No. 63 / 779,784, filed March 28, 2025, which is incorporated by reference in its entirety.TECHNICAL FIELD

[0002] The present disclosure relates to integrating Large Language Models (LLMs) with healthcare sensor data to enhance personalized user interactions. More specifically, it involves using sensed physiological and environmental data to generate refined LLM prompts and tailored context-aware, LLM responses for improved context-aware and empathetic healthcare recommendations.BACKGROUND

[0003] Large Language Models (LLMs), such as GPT, have revolutionized the field of artificial intelligence (Al) by providing powerful tools for data analysis and natural language processing (NLP). These models are capable of understanding and generating human-like text, making LLMs useful in a wide range of applications, from chatbots to content creation. LLMs leverage vast amounts of data and sophisticated algorithms to predict and generate text based on user inputs, offering impressive capabilities in terms of language understanding and generation.

[0004] In the healthcare domain, specific LLMs have been developed to utilize sensing data for predicting health conditions. These models analyze data from various sensors, such as wearable devices, to assess a user's health status, including indicators, such as heart rate, sleep quality, and activity levels. Healthcare-specific LLMs aim to provide insights into potential health issues, such as fatigue, anxiety, or chronic conditions, by interpreting physiological data and offering diagnostic predictions.

[0005] However, existing LLMs face significant shortcomings in the context of healthcare. One limitation is a lack of personalization, as LLMs do not consider the user's emotional or physiological state when generating responses, leading to generic and potentially unsuitable076333-1064 / 06906 PATENTinteractions. For instance, an existing LLM might suggest physically demanding activities to a user who has had poor sleep, without recognizing the need for rest and relaxation.

[0006] Furthermore, current healthcare-specific LLMs focus primarily on predicting health conditions rather than leveraging certain healthcare data to refine user queries or provide personalized suggestions. While these models can diagnose potential health issues, existing LLMs do not integrate environmental or physiological data to tailor responses, missing the opportunity to offer empathetic and context-aware interactions. This absence of prompt engineering based on sensed data limits the effectiveness of LLMs in delivering personalized healthcare recommendations.

[0007] Consequently, existing approaches to LLMs and healthcare-specific LLMs exhibit a gap in providing context-aware and empathetic responses, highlighting the need for advancements that integrate physiological and environmental data to enhance personalization and user experience.SUMMARY

[0008] Disclosed herein are systems and methods capable of addressing the abovedescribed shortcomings and may provide any number of additional or alternative benefits and advantages. Embodiments described herein provide for an advanced Al system integrating Large Language Models (LLMs) with healthcare sensor data to enhance personalized user interactions. The software programming executed by a user device obtains sensor data of various categories, such as activity level, stress level, and sleep quality, and generates a sensor aggregation file (or other data format for a user profile summary) with severity metrics (sometimes referred to as severity scores) for each category. The system obtains a user-entered LLM prompt and performs a first call to an LLM to generate a refined-LLM prompt based on the sensor aggregation file, and then obtains a tailored response to the refined-LLM prompt by performing a second call to the LLM.

[0009] Embodiments may include a processor-implemented method for implementing improved LLM prompts using healthcare sensor data. The method includes: obtaining, by at least one processor, sensor data of one or more sensors including a plurality of sensor records for a user, where the plurality of sensor records includes a plurality of subsets of sensor data corresponding076333-1064 / 06906 PATENTto a plurality of sensor data categories; receiving, by the at least one processor, a user-entered LLM prompt via a user interface of a user device having the at least one processor; for each sensor data category, determining, by the at least one processor, a severity metric or score for the sensor data category based upon the sensor data of sensor records of the subset of sensor records of the sensor data category; generating, by the at least one processor, a sensor aggregation file containing each subset of sensor data records and each severity metric or score for each category; instructing, by the at least one processor, a first call to an LLM to generate a refined LLM prompt based upon the user-entered LLM prompt and the sensor aggregation file using the LLM; instructing, by the at least one processor, a second call to the LLM to generate an LLM response based upon the refined LLM prompt and the sensor aggregation file using the LLM; and generating, by the at least one processor, a user interface output including the LLM response for display at the user interface of the user device having.

[0010] The LLM may be installed on the user device and executed by the at least one processor of the user device. The LLM may be installed on a remote computing device that is remote to the user device having the at least one processor.

[0011] The method may include transmitting, by the at least one processor, the user-entered LLM prompt and the sensor aggregation file to the remote computing device via one or more networks; and receiving, by the at least one processor, the refined LLM prompt from the remote computing device via the one or more networks.

[0012] The method may include transmitting, by the at least one processor, the refined LLM prompt and the sensor aggregation file to the remote computing device via one or more networks; and receiving, by the at least one processor, the refined LLM prompt from the remote computing device via the one or more networks.

[0013] The method may include determining, by the at least one processor, the sensor data category of a sensor data record based upon a preconfigured mapping between one or more types of sensor data of the sensor data record and the sensor data category.

[0014] When determining the severity metric or score for the sensor data category, the method may include generating, by the at least one processor, the severity metric or score based076333-1064 / 06906 PATENTupon a current sensor data record and each sensor data record of the subset of sensor data records having a timestamp satisfying a preconfigured timeframe interval.

[0015] The method may include: determining, by the at least one processor, one or more severity level thresholds for the category; and determining, by the at least one processor, a severity level of the category based upon comparing the severity metric or score against the one or more severity level thresholds.

[0016] The method may include determining, by the at least one processor, a query refinement goal for the sensor data category based upon the severity metric or score for the sensor data category. The at least one processor obtains the refined LLM prompt further based upon the query refinement goal.

[0017] The sensor data categories of the sensor data may include at least one of activity level, stress level, or sleep quality.

[0018] Embodiments may include a system for implementing improved LLM prompts using healthcare sensor data. The system may include a user device having a at least one processor configured to: obtain sensor data of one or more sensors including a plurality of sensor records, the plurality of sensor records including a plurality of subsets of sensor data corresponding to a plurality of sensor data categories; receive a user-entered LLM prompt via a user interface of the user device having the at least one processor; for each sensor data category, determine a severity metric or (sometimes referred to as a severity score) for the sensor data category based upon the sensor data of sensor records of the subset of sensor records of the sensor data category; and generate a sensor aggregation file containing each subset of sensor data records and each severity metric or score for each category; instruct a first call to an LLM to generate a refined LLM prompt based upon the user-entered LLM prompt and the sensor aggregation file using the LLM; instruct a second call to the LLM to generate an LLM response based upon the refined LLM prompt and the sensor aggregation file using the LLM; and generate a user interface output including the LLM response for display at the user interface of the user device.

[0019] The LLM may be installed on the user device and executed by the at least one processor of the user device. The LLM may be installed on a remote computing device that is remote to the user device having the at least one processor.076333-1064 / 06906 PATENT

[0020] The at least one processor may be further configured to: transmit the user-entered LLM prompt and the sensor aggregation file to the remote computing device via one or more networks; and receive the refined LLM prompt from the remote computing device via the one or more networks.

[0021] The at least one processor may be further configured to: transmit the refined LLM prompt and the sensor aggregation file to the remote computing device via one or more networks; and receive the refined LLM prompt from the remote computing device via the one or more networks.

[0022] The at least one processor may be further configured to determine the sensor data category of a sensor data record based upon a preconfigured mapping between one or more types of sensor data of the sensor data record and the sensor data category.

[0023] When determining the severity metric or score for the sensor data category, the at least one processor may be further configured to generate the severity metric or score based upon a current sensor data record and each sensor data record of the subset of sensor data records having a timestamp satisfying a preconfigured timeframe interval.

[0024] The at least one processor may be further configured to: determine one or more severity level thresholds for the category; and determine a severity level of the category based upon comparing the severity metric or score against the one or more severity level thresholds.

[0025] The at least one processor may be further configured to determine a query refinement goal for the sensor data category based upon the severity metric or score for the sensor data category. The at least one processor may obtain the refined LLM prompt further based upon the query refinement goal.

[0026] The sensor data categories of the sensor data may include at least one of activity level, stress level, or sleep quality.

[0027] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are intended to provide further explanation of the invention as claimed.076333-1064 / 06906 PATENTBRIEF DESCRIPTION OF THE DRAWINGS

[0028] The present disclosure can be better understood by referring to the following figures. The components in the figures are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the disclosure. In the figures, reference numerals designate corresponding parts throughout the different views.

[0029] FIG. 1 shows components of a system for implementing an LLM to generate improved, refined LLM prompts and response outputs using various types of sensor data, according to certain embodiments.

[0030] FIG. 2 shows dataflow amongst executable software components and hardware components of a mobile processing device, according to certain embodiments.

[0031] FIG. 3 shows a structured JSON file as an example output of the sensing results processing engine, according to certain embodiments, according to certain embodiments.

[0032] FIG. 4 is a flowchart showing operations of a processor-implemented method for implementing an LLM to generate improved outputs using various types of sensor data, including health-related data and health-related sensors, using computer-refined LLM prompts, according to certain embodiments.DETAILED DESCRIPTION

[0033] Reference will now be made to the illustrative embodiments illustrated in the drawings, and specific language will be used here to describe the same. It will nevertheless be understood that no limitation of the scope of the invention is thereby intended. Alterations and further modifications of the inventive features illustrated here, and additional applications of the principles of the inventions as illustrated here, which would occur to a person skilled in the relevant art and having possession of this disclosure, are to be considered within the scope of the invention.

[0034] FIG. 1 shows components of a system 100 for implementing an LLM 122 to generate improved outputs using various types of sensor data. The system 100 includes a mobile processing device 101 and sensing devices 102a-102b (generally referred to as sensing devices 102). Optionally, the system 100 includes an analytics server 104 and analytics database 106.076333-1064 / 06906 PATENT

[0035] The mobile processing device 101 functions as a central hub for executing software programming designed to process the user inputs via a user interface and sensory data via the sensing devices 102 or onboard sensors. The mobile processing device 101 interacts with or executes software programming for one or more machine-learning models, including software for rewriting a user-entered LLM prompt and the LLM 122 for generating responses, based upon various operations integrating various types of sensor data to provide context-aware and empathetic responses relative to the user’s physical and physiological state.

[0036] The mobile processing device 101 includes any electronic device capable of performing the various processes and tasks described herein. The mobile processing device 101 includes processors (e.g., CPU, GPU) and non-transitory machine-readable storage media as storage memory for storing executable software instructions and various types of data (e.g., various types of sensor data records), among other types of hardware components. For instance, in some cases, the mobile processing device 101 may include hardware and software components that perform networked, wired or wireless communications with external sensing devices 102 that generate or otherwise capture input sensor data. In some cases, the mobile processing device 101 includes hardware and software components of onboard sensors that are integrated within the mobile processing device 101 that generate or otherwise capture sensor data. Non-limiting examples of the mobile processing device 101 include smartphones, mobile tablet devices, and smartwatches 102b, among other types of electronic devices.

[0037] As an example, the mobile processing device 101 may include a smartphone equipped with processors, memory, and a variety of onboard sensors, such as accelerometers, gyroscopes, heart rate monitors, and GPS, among others. The mobile processing device 101 may obtain sensor data from the onboard sensors, which includes accelerometers measuring acceleration forces and detecting movement and orientation, gyroscopes providing information on device orientation and angular velocity, heart rate monitors tracking heart rate and providing insights into physiological states, and GPS providing location data and enabling tracking of movements and activities. As another example, the mobile processing device 101 may include a tablet device having a larger screen and more powerful hardware processors and memory for running complex LLM 122 software and handling extensive sensory data inputs. As another example, the mobile processing device 101 may include a smartwatch with processors and076333-1064 / 06906 PATENTintegrated sensors, can continuously collect and transmit user data for real-time processing and response generation to another device, such as an analytics server 104 that executes the LLM 122 software programming described herein.

[0038] The mobile processing device 101 may perform preprocessing operations on the sensor data. The preprocessing on collected sensor data includes, for example, data normalization for uniformity, noise reduction to filter out irrelevant or erroneous data, and feature extraction to identify metrics relevant to well-being and context. Real-time data processing allows the LLM 122 to generate immediate and contextually appropriate responses, continuously collecting and analyzing up-to-date sensory data indicating physical and physiological state. Collected sensor data may be stored in a non-transitory storage medium in a structured format, available for further processing and analysis. Historical sensor data can be retrieved to configure or train machinelearning models to identify long-term trends and patterns. A query rewriting engine refines input queries based on sensor-derived context, leading to more accurate and personalized responses. The user interface engine enables interactions with the LLM 122, including displaying responses, accepting inputs, and providing feedback mechanisms to improve accuracy and relevance of outputs.

[0039] The sensing devices 102 may include any type of electronic device having hardware and software components designed to capture a variety of sensor data, enabling the generation of personalized and context-aware responses by the software programming of the mobile processing device 101 and the LLM 122. These sensing devices 102 include a wide range of wearable and mobile sensors capable of continuously monitoring the user’s physical and physiological states. These sensing devices 102 are compact, portable, and equipped with high-precision sensors that interface with the mobile processing device 101 through wired or wireless communication channels.

[0040] Functionally, the sensing devices 102 collect real-time data on different aspects of the user’ s activities and well-being. The collected sensor data includes metrics or other information in various categories, such as physical activities, physiological metrics, and environmental conditions. The software programming of the mobile processing device 101 process and categorize user sensory records containing the sensory data, which the mobile processing device 101 feeds to the machine-learning models and LLM 122 to infer user state context and generate a refined query,076333-1064 / 06906 PATENTwhich the mobile processing device 101 feeds as the rewritten LLM prompt to the ELM 122 that generates an LLM-generated response.

[0041] The sensing devices 102 encompass a variety of wearables and mobile devices having sensors, such as accelerometers, gyroscopes, heart rate monitors, GPS, blood pressure monitors 102a, and smartwatches 102b. Each type of sensing device 102 captures specific sensor data that provides insights into different aspects of the user’s condition. For example, Accelerometers measure acceleration forces to detect movement and orientation, while gyroscopes provide information on device orientation and angular velocity. Heart rate monitors track the user’ s heart rate, offering valuable data on their physiological state, and GPS devices provide location data for tracking movements and activities.

[0042] The software programming of the mobile processing device 101 categorize the types of sensor data received from these sensing devices 102 into several high-level descriptive categories, which facilitate further analysis and contextual understanding. An “activity level” category includes data such as the number of steps taken, distance traveled, and types of exercises performed. In some cases, metrics or information are within this category, such as calories burned and indicators of different levels of activity (lightly active, moderately active, very active). A “sleep quality” category encompasses sleep duration, sleep efficiency, and indicators of sleep disturbances, providing insights into the user’s sleep patterns and overall sleep quality. A “stress level” category includes information or metrics such as stress scores, heart rate variability, and participation in mindfulness sessions. The stress level category may also include physiological indicators, such as resting heart rate and non-REM heart rate, among other indicators of the user’s stress levels.

[0043] As an example, a blood pressure monitor 102a is an example of a sensing device 120 that monitors cardiovascular health. The blood pressure monitors 102a typically includes an inflatable cuff that wraps around the user's upper arm and a digital display that shows systolic and diastolic blood pressure readings. The blood pressure monitor 102a generates sensory data, which includes real-time blood pressure readings, and transmits this sensory data to the mobile processing device 101 via a wired or wireless connection. The software programming of the mobile processing device 101 (or analytics server 104) determines the category and analyzes the user’s cardiovascular health, which the machine-learning models and LLM 122 may reference and analyze to provide076333-1064 / 06906 PATENTpersonalized health recommendations or responses to LLM prompts, such as dietary adjustments or stress management techniques.

[0044] As another example, a smartwatch 102b is another example of sensing devices 102. Equipped with a variety of sensors, such as accelerometers, gyroscopes, heart rate monitors, and GPS, the smartwatch 102b continuously collects a broad spectrum of sensory data. For instance, the sensors and software of the smartwatch 102b tracks the user’s steps, distance traveled, heart rate, and location throughout the day, among other types of sensory data. The smartwatch 102b may also monitor sleep patterns by detecting movement and heart rate during sleep. The smartwatch 102b transmits this sensory data to the mobile processing device 101 via a wired or wireless connection. The software programming of the mobile processing device 101 (or analytics server 104) determines the category of the user’s sensory data as physical activity, physiological state, and sleep quality, which the machine-learning models and LLM 122 may reference and analyze to provide personalized health recommendations or responses to LLM prompts.

[0045] The analytics server 104 may be any electronic computing device that includes software and hardware components capable of performing the various processes and operations described herein. In some embodiments, the analytics server 104 performs operations for training an LLM 122. This involves processing vast amounts of sensory data and user interaction records (e g., LLM prompts) to fine-tune the LLM 122 responses to various contexts and queries. Once the LLM 122 is thoroughly trained, the LLM 122 is then installed on the mobile processing device 101 (or other type of end-user computing device), enabling the mobile processing device 101 to leverage the trained LLM 122 locally, such that the mobile processing device 101 executes the operations of the LLM 122 locally.

[0046] In some embodiments, the LLM 122 is installed and executed at a remote computing device that is remote to the mobile processing device 101 (or other end-user device), such as the analytics server 104. The analytics server 104 executes the LLM 122 using various types of sensory data and user interactions transmitted from the mobile processing device 101 over one or more networks. The mobile processing device 101 collects sensory data from the sensing devices 102 and user inputs via the user interface, which the mobile processing device 101 transmits to the analytics server 104 via wired or wireless networks using one or more APIs. The analytics server 104 executes the LLM 122 to generate context-aware and personalized responses076333-1064 / 06906 PATENTusing a refined LLM prompt and transmits an LLM-generated response to the mobile processing device 101 for display at the user interface.

[0047] FIG. 2 shows dataflow amongst executable software components and hardware components of a mobile processing device 200. The mobile processing device 200 includes a processor that executes software programming that includes a sensing results processing engine 202, a query rewriting query rewriting query rewriting engine 204, and a user interface engine 206.

[0048] The mobile processing device 200 includes processors (e.g., CPU, GPU), and non-transitory machine-readable storage media for storing executable software instructions and various types of data (e.g., various types of sensor data records), among other types of hardware components. For instance, in some cases, the mobile processing device 200 may include hardware and software components that perform networked, wired or wireless communications with external sensing devices (e.g., sensing devices 102) that generate or otherwise capture input sensor data. In some cases, the processing device 200 includes hardware and software components of onboard sensors that are integrated within the mobile processing device 200 that generate or otherwise capture sensor data.

[0049] The sensing results processing engine 202 includes a data collection engine 210 and a data categorization engine 212. The sensing results processing engine 202 handles and interprets sensory data from the sensing devices, which may include various types of wearable and mobile devices.

[0050] The data collection engine 210 of the sensing results processing engine 202 is responsible for gathering data from various sensors embedded in wearable and mobile devices, such as accelerometers, gyroscopes, heart rate monitors, and GPS. The data collection engine 210 collects data on physical activities, physiological metrics, and environmental factors.

[0051] The data collection engine 210 collects, receives, or otherwise obtains sensory data as sensor records from various embedded sensors of, for example, wearable devices and mobile devices. The sensors include, for example, accelerometers, gyroscopes, heart rate monitors, and GPS. Accelerometers measure acceleration forces and detect movement and orientation. Gyroscopes provide information on the device's orientation and angular velocity. Heart rate076333-1064 / 06906 PATENTmonitors track the user's heart rate and provide insights into physiological states. GPS provides location data and enables tracking of movements and activities.

[0052] The data collection engine 210 collects sensor data on physical activities performed by the user. The sensor data includes, for example, a number of steps taken throughout the day, distance traveled during different activities, and types of exercises performed, such as walking, running, or biking. The data collection engine 210 also gathers sensor data representing physiological metrics, including heart rate variability, resting heart rate, and caloric expenditure. Additionally, the sensor data may represent environmental factors that may affect well-being, including ambient temperature, humidity levels, and air quality, among others.

[0053] The data collection engine 210 interfaces with the wearable and mobile devices, allowing real-time data collection. The data collection engine 210 may perform continuous data collection for obtaining and analyzing up-to-date sensor data indicating the user's physical and physiological state. In some embodiments, the collected sensor data may be stored in the non-transitory storage medium, in the form of a structured format available for further processing and analysis by downstream operations of the sensing results processing engine 202. In some implementations, the data collection engine 210 may store and retrieve historical sensor data for configuring or training the machine-learning models of the machine-learning architecture to identify long-term trends and patterns in behavior and health metrics.

[0054] In some embodiments, the data collection engine 210 performs preliminary preprocessing on the collected data to maintain quality and relevance. These preprocessing operations may include, for example, data normalization for uniformity, noise reduction to filter out irrelevant or erroneous data, and feature extraction to identify key metrics relevant to wellbeing. The data collection engine 210 aggregates diverse sensory data, maintains quality, and makes data available for further analysis by other components of the software programming.

[0055] The data categorization engine 212 of the sensing results processing engine 202 organizes the collected sensor data into sensor-data categories, such as activity level, sleep quality, and stress level. The data categorization engine 212 includes software programming of one or more machine-learning models for sensor data interpretation, including a sensor data classifier layer. The sensor data classifier of the data categorization engine 212 is programmed and trained to076333-1064 / 06906 PATENTanalyze the categorized sensor data and infer, for example, a user’s current physical and physiological states. The data categorization engine 212 may include layers of the machinelearning model and predefined functions programed and / or trained to generate meaningful insights about the user’s condition.

[0056] The machine-learning models in the data categorization engine 212 are trained using sensor data to interpret and classify this data into meaningful user context categories. A computing device (e.g., sensing devices 102, analytics server 104) trains the machine-learning models to recognize and analyze patterns from amongst the various types of sensory data and / or categories (e.g., physical activity, sleep quality, stress level). These models include layers and predefined functions that are programmed to generate insights into the user’s physical and physiological states. The training process involves feeding historical sensor data into the models, allowing the machine-learning models to learn and identify long-term trends and patterns in behavior and health metrics. The models of the data categorization engine 212 (or query rewriting engine 204) perform functions, such as categorization of sensor data, infer current user states, and generate relevant insights that are utilized to provide personalized, context-aware responses and recommendations.

[0057] The machine-learning models of the data categorization engine 212 process the sensory data to create or compute indices for each sensory category, including activity level, sleep quality, and stress level. Based on these indices, the sensory data classifier may classify the sensor data in each sensor data record as a type of sensory category. For activity level, the models analyze data such as calories burned, steps taken, and minutes of activity to determine overall physical activity. Sleep quality is assessed by examining factors like sleep duration and efficiency, providing an evaluation of restfulness. Stress level is measured by reviewing indicators such as heart rate variability, scores from stress-monitoring tools, and mindfulness session participation. This data is categorized into levels like low, medium, and high based on deviations from typical patterns, helping to identify significant changes in physical and mental states.

[0058] In some embodiments, the data categorization engine 212 uses predefined rules to categorize the sensor data of a sensor data record into a category (e.g., activity level, sleep quality, stress level). The data categorization engine 212 includes one or more preconfigured mappings,076333-1064 / 06906 PATENTincluding a preconfigured mapping for each type of sensor data indicating a corresponding category for the type of sensor data.

[0059] In some embodiments, the data categorization engine 212 performs statistical analysis on the sensor data records of the user to determine patterns. For example, the data categorization engine 212 calculates an average and standard deviation for each metric over a specified period (e.g., a week). Based on these calculations, the data categorization engine 212 labels the data according to a severity metric (sometimes referred to as a severity score) or severity level for the physical activity category (e.g., “high,” “medium,” “low”) corresponding to one or more severity metric thresholds.

[0060] For example, if the user’s walking distance is significantly above an average (e.g., more than one standard deviation above), then the data categorization engine 212 determines or labels the physical activity category of the user as a “high” activity level. As another example, if the user’s REM sleep duration is significantly below an average (e.g., more than one standard deviation below), then the data categorization engine 212 determines or labels the sleep quality category for the user as a “low” sleep quality.

[0061] The data categorization engine 212 may determine or compute personalized thresholds for the user. The data categorization engine 212 compares or otherwise algorithmically evaluates the current sensor data inputs against the user’s historical sensor data to determine the categories and / or severity levels for the categories.

[0062] For instance, the data categorization engine 212 identifies a subset of sensor data records having sensor data in a category and extracts values of sensor data fields or features from the sensor data records having a given timestamp in a timeframe interval (e.g., prior 7 days), where the timestamp satisfies one or more preconfigured timeframe intervals. At a given preconfigured timeframe interval, the data categorization engine 212 computes a moving average value using the values of the sensor data fields, which the data categorization engine 212 treats as a severity metric (sometimes referred to as a severity score) or a severity level (e.g., high, medium, low) for the category. The data categorization engine 212 may also compute one or more standard deviations using the values of the sensor data fields, which the data categorization engine 212 treats as one or more severity level thresholds for the category. For example, the data categorization engine 212076333-1064 / 06906 PATENTdetermines a category severity metric or score representing an average value of one or more particular data fields or a category using sensor data values in the subset of a data records having timestamps in the preceding week. The data categorization engine 212 also determines a “high” threshold and a “low” threshold representing a standard deviation above the average value and a standard deviation below the average value.

[0063] Based upon the corresponding values (e.g., severity score, severity level) computed by the data categorization engine 212 for each category, the data categorization engine 212 classifies the activity level category, sleep quality category, and stress level category. For instance, the data categorization engine 212 determines that a category or data value is classified as ’high’ if the severity metric (or other types of sensor data values) for the category exceeds a “high” threshold. Likewise, the data categorization engine 212 determines that a category or data value is classified as “low” if the severity metric (or other types of sensor data values) for the category exceeds a “low” threshold.

[0064] The sensing results processing engine 202 may execute the preprocessing operations using the data collection engine 210, before the data categorization engine 212, and / or before the sensing results processing engine 202 feeds the sensor data inputs to the LLM 222 of the query rewriting query rewriting engine 204. These preprocessing operations of the sensing results processing engine 202 may include, for example, data normalization, noise reduction, feature extraction, and / or embedding extraction as feature vectors using extracted features, among other types of preprocessing operations.

[0065] As shown in FIG. 3, the sensing results processing engine 202 generates and outputs the processed sensory data for a sensor aggregation file as a structured JSON file 300. For each data category, the JSON file 300 includes a current value, a calculated average, a standard deviation or threshold, and a severity metric (sometimes referred to as a severity score) or severity classification level (e.g., “high,” “moderate,” “low”). In some implementations, this JSON file 300 is updated daily and serves as an input for the query rewriting engine 204 or may be generated or updated by the query rewriting engine 204. These processes of the sensing results processing engine 202 allows the components of the software programming of the mobile processing device 200 (e.g., query rewriting engine 204) to generate the sensor aggregation file (or other data format) including or otherwise representing a comprehensive user profile of the user sensor data and076333-1064 / 06906 PATENTinformation indicating the user’s physical and mental state, enabling more the LLM 222 to generate personalized, context-aware, and empathetic final output responses 229.

[0066] Turning back to FIG. 2, the query rewriting engine 204 may include the LLM 222 programmed and trained for generating textual responses to user-generated or user-entered LLM prompts within user input queries 226, where the user interface engine 206 displays as the final output response 229 at a user interface of the mobile processing device 200. The query rewriting engine 204 may further include the user profile analyzer 223, the query refinement engine 224, and the refined query generator 225 that generates a refined query 227 containing a refined LLM prompt, as a refined or improved version of the original user-entered LLM prompt. In particular, the various components of the query rewriting engine 204 receive the original user-entered LLM prompt of the original user input query 226, analyze the user profile information of the sensor aggregation file, and generate the refined LLM prompt for the refined query 227 as the refined version of the original user input query 226.

[0067] The query refinement engine 224 is designed to obtain and analyze the processed sensor data from the sensing results processing engine 202 to refine user queries from the original user input query 226 so that the responses generated by the LLM 222 are contextually aware and relevant with the user’s current and historical physical and mental state.

[0068] The LLM 222 is programmed and trained to generate textual responses to user input queries 226 displayed on the user interface of the mobile processing device 200. This process involves several steps, beginning with the query rewriting engine 204 obtaining and analyzing the processed sensory data from the sensing results processing engine 202. Based on this analysis, the query refinement engine 224 determines a primary query refinement goal aligned with the specific context of the user. This goal is selected from a predefined set of potential goals, such as relieving stress, promoting physical wellness, improving productivity, enhancing mental clarity, and providing emotional support. After determining the refinement goal, the refined query generator 225 rewrites the original user input query 226 to generate a refined query 227. This refined query incorporates the selected refinement goal and contextual profile data from the sensor aggregation file. The LLM 222 then uses this refined query to generate final output responses 229 that are contextually aware and relevant to both the user's original input and current state, thereby providing personalized and empathetic responses.076333-1064 / 06906 PATENT

[0069] The ELM 222 may be trained or tuned using example-based training or tuning, in which specific examples of, for example, original LLM prompts of user input queries 226, refined LLM prompts of refined queries 227, responses 229, and input sensor data. The programming and machine-learning architecture (e.g., neural network architecture) of the LLM 222 learns and updates the neural network architecture from the specific examples according to desired behaviors or outputs, as indicated by, for example, user-generated feedback or training labels. This method leverages few-shot or zero-shot learning techniques, where only a small number of examples or even none are used to guide training processes of the LLM 222. The example-based training helps the LLM 222 generalize and tune certain parameters based upon the example inputs.

[0070] The mobile processing device 200 or remote computing devices (e.g., analytics server 104) use example-based tuning to guide the LLM 222 in rewriting the LLM prompt of the original user input query 226 to generate the refined query 227 containing the refined LLM prompt. The software programming for training the LLM 222 provides the examples indicating to the LLM 222 how user input queries 226 should be rewritten based on various input data or values, such as different physiological states. For instance, if the user has “high” activity levels and “low” sleep quality, then the LLM 222 is trained or tuned to suggest rewriting the user-entered LLM prompt of the user input query 226 and / or final output response 229 to prioritize relaxing activities.

[0071] In some embodiments, the LLM 222 software programming and trained models are stored, hosted, and executed in the local memory of the mobile processing device 200. In such embodiments, the mobile processing device 200 receives the original user input query 226 containing the user-entered LLM prompt via the user interface engine 206. The mobile processing device 200 then invokes and executes the LLM 222 using the original user input query 226 containing the user-entered LLM prompt and the categorized or labeled sensor data from the sensing results processing engine 202. Using the provided examples, the LLM 222 processes the original LLM prompt of the original input query 226 and the sensor data to generate the rewritten, refined LLM prompt of the refined query 227. The mobile processing device 200 may again invoke and execute the LLM 222 using the rewritten, refined prompt refined query 227 to generate the final output response 229 for display by the user interface engine 206 at the user interface of the mobile processing device 200.076333-1064 / 06906 PATENT

[0072] In some embodiments, the LLM 222 software programming and trained models are stored, hosted, and executed at a remote computing device (e.g., analytics server 104). In such embodiments, the mobile processing device 200 initiates and executes a first API call to the remote computing device hosting the LLM 222, which the mobile processing device 200 uses to transmit and submit an original user input query 226 containing the user-entered LLM prompt and the categorized or labeled sensor data from the sensing results processing engine 202, to the remote computing device. Using the provided examples, the LLM 222 processes the original prompt of the original user input query 226 and the sensor data to generate the rewritten, refined prompt of the refined query 227. The mobile processing device 200 may then initiate a second API call to the LLM 222, submitting the rewritten prompt of the refined query 227 to the LLM 222 to generate the final output response 229. The mobile processing device 200 then receives the final output response 229 from the LLM 222 of the remote computing device.

[0073] The user profile analyzer 223 obtains and analyzes the sensor data records of the user’s sensor aggregation file, having various sensor data categories, such as physical activity levels, sleep quality metrics, and stress levels. By analyzing this data, the user profile analyzer 223 is able to gather and synthesize critical context information for the user and user state.

[0074] The user profile analyzer 223 generates a summary of the user’s sensor aggregation file based upon, for example, the categorized and labeled sensor data. This summary includes various types of values or metrics computed using sensor data fields of the sensor data records in the various categories. The user profile analyzer 223 generates the sensor aggregation file or report that includes, for example, aggregated sensor data, statistical summaries, and assigned labels for the subsets of data records for the categories.

[0075] After the user profile analyzer 223 generates the sensor aggregation file containing the various types of user sensor data and profile data, the rewriting engine 204 executes the query refinement engine 224 to determine a primary query refinement goal. The query refinement engine 224 includes machine-learning models trained to determine a query refinement goal based upon and aligned with the user’s specific context. In some implementations, the rewriting engine 204 includes a pre-defined a set of potential goals based on common user needs and psychological principles (e.g., relieving stress, promoting physical wellness, improving productivity, enhancing mental clarity, providing emotional support). The query refinement engine 224 uses the contextual076333-1064 / 06906 PATENTinformation from the analysis of the user sensor and profile data by the user profile analyzer 223 to select one or more appropriate refinement goals.

[0076] The query refinement engine 224 reviews the sensor aggregation file, which includes categorized and labeled sensor data for the various categories, such as activity level, sleep quality, and stress levels. The user profile analyzer 223 or the query refinement engine 224 assesses or determines the user’s current state by analyzing the most recent sensor data and comparing the current sensor data against the user’s historical data. Based on the current state assessment, the user profile analyzer 223 or query refinement engine 224 infers the user’s needs. For example, if the user has a “high” activity level and low sleep quality, then the query refinement engine 224 may determine or infer that the user needs rest and relaxation to increase or improve values of the types of sensor data fields that map to the sleep quality category. As another example, if the user has a “low” activity level and “high” sleep quality, then the query refinement engine 224 may determine or infer that the user needs more physical activity to increase or improve values of the types of sensor data fields that map to the physical activity category. In this way, the query refinement engine 224 determines or selects a refinement goal based on the determined or inferred needs of the user. This refinement goal guides the query rewriting engine 204 in rewriting the user-entered LLM prompt of the user-entered query 226 to be more personalized to the user.

[0077] In some embodiments, the query refinement engine 224 may be programmed or trained to select the refinement goal using various machine-learning techniques. For example, the query refinement engine 224 may be trained to select a refinement goal using example-based training for the LLM 222, according to a set of few-shot prompting examples to help the LLM 222 understand the query refinement goals and prioritize the user’s current needs based on the identified context.

[0078] After the query refinement engine 224 determines and selects the refinement goal from the set of pre-defined query refinement goals, the software programming executes the refined query generator 225 to rewrite the original user input query 226 and generate the refined query 227. The refined query generator 225 incorporates the selected query refinement goal and the contextual sensor and profile data from the user’s sensor aggregation file, such that the refined query 227 and a final output response 229 is tailored to both the user’s original user input query 226 and also the user’s current state. For example, the user submits an original user input query076333-1064 / 06906 PATENT226 initially focused on completing a professional task and the refined query generator 225 generates a refined query 227 reframed to consider stress reduction, when the data categorization engine 212 and the user profile analyzer 223 indicates high stress levels in the sensory data records of the sensor aggregation file.

[0079] The user interface engine 206, as software programming of the mobile processing device 200, is programmed to generate and update interactive user interfaces for displaying output information to the user (e.g., refined query 227, final output response 229) and obtaining user inputs (e.g., original user input query 226). For instance, the user interface displays an interactive user interface that allows the user to enter the original user input query 226 as an LLM prompt for the LLM 222. This original user input query 226 is captured and fed to the software programming of the rewriting engine 204, which analyzes and enhances the original user input query 226 based on contextual information derived from the user’s sensor aggregation file by the data categorization engine 212 and user profile analyzer 223. The refined query 227 is a refined, computer-generated version of the original user input query 226, tailored to better address the user’s needs, and the user interface engine 206 displays the refined query 227 on the user interface and the rewriting engine 204 feeds the refined query 227 to the LLM 222 as a computer-generated LLM prompt. The LLM 222 generates the final output response 229 using the refined query 227 and the user interface engine 206 displays the final output response 229 at the user interface for the user.

[0080] In the example depicted in FIG. 2, a user has a professional presentation scheduled for the following day and seeks guidance from the software programming. The user enters the original user input query 226 at the user interface generated by the user interface engine 206 (e.g., “How should I prepare for my presentation tomorrow?”). The sensing results processing engine 202 receives, processes, and categorizes input sensor data records. The data collection engine 210 of the sensing results processing engine 202 receives or retrieves relevant sensing data as sensor data records from the user’s wearable devices, onboard sensors of the mobile processing device 200, or from the storage medium. For instance, the data collection engine 210 extracts relevant sensing data, including the user’s heart rate, physical activity levels, and recent sleep patterns, from a wearable device. The data categorization engine 212 determines that the sensing data records076333-1064 / 06906 PATENTreveals that the user had moderate physical activity today and experiencing elevated stress, as indicated by a persistently high heart rate variability and reduced sleep duration.

[0081] Continuing with this example, the sensing results processing engine 202 feeds the categorized sensor data records to the query rewriting engine 204, where the rewriting engine 204 evaluates the user’ s physical and mental states. The user profile analyzer 223 and query refinement engine 224 identifies and selects a predefined refinement goal that aligns with the patient needs, and the refined query generator 225 uses the original user input query 226 to generate the refined query 227. The LLM 222 uses the refined query 227 to generate the final output response 229. For instance, the user profile analyzer 223 receives the categorized sensor data from the data categorization engine 212, and the query refinement engine 224 identifies “relieve stress” as a query refinement goal. The refined query generator 225 then rewrites the user’s original user input query 226 to generate the refined query 227 (e.g., “Considering my moderate activity levels, elevated stress, and low sleep quality, how should I prepare for my presentation to ensure it goes smoothly and I can manage my stress effectively?”). This refined query 227 is sent as the LLM prompt to the LLM 222 (or API to a server having the LLM 222), which generates the final output response 229 that addresses the user’s presentation preparation and offers strategies to manage the user’s stress. The user interface engine 206 then generates or updates the user interface to display the final output response 229.

[0082] FIG. 4 is a flowchart showing operations of a processor-implemented method 400 for implementing a Large Language Model (LLM) to generate improved outputs using various types of sensor data, including health-related or healthcare data and health-related or healthcare sensors, using computer-refined LLM prompts. The operations of the method 400 are described as being executed by at least one processor of a user device, such as a mobile device (e.g., smartphone, tablet device).

[0083] At operation 410, the at least one processor obtains healthcare sensor data of one or more sensors including a plurality of sensor records, where the sensor data and sensor records may be generated by the one or more sensors and received or retrieved from the one or more sensors, and / or may be received or retrieved from a database containing the sensor records. The sensor records include a plurality of subsets of sensor data corresponding to a plurality of sensor data categories. In some cases, each of the sensor records or subsets of the sensor data may be076333-1064 / 06906 PATENTclassified into respective sensor data categories. At operation 420, the processor receives a user-entered LLM prompt via a user interface of the user device having the processor.

[0084] At operation 430, for each sensor data category, the at least one processor determines a severity metric (sometimes referred to as a severity score) for the sensor data category based upon the sensor data of each sensor record of the subset of sensor records of the sensor data category. At operation 440, the at least one processor generates a sensor aggregation file (or other format of a user profile summary) containing each subset of sensor data records and each severity metric for each category.

[0085] At operation 450, the at least one processor instructs or otherwise sends a first call to an LLM to invoke or execute one or more inference operations (e.g., a first inference operation) to generate a refined LLM prompt based upon the user-generated or user-entered LLM prompt and the sensor aggregation file using the LLM. At operation 460, the at least one processor instructs or otherwise sends a second call to the LLM to invoke or execute one or more further inference operations (e g., a second inference operation) to generate an LLM response based upon the refined LLM prompt and the sensor aggregation file.

[0086] At operation 470, the at least one processor generates a user interface output for display at the user interface of a user device having the at least one processor, the user interface output including the context-aware LLM response as obtained from the LLM.

[0087] The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

[0088] Embodiments implemented in computer software may be implemented in software, firmware, middleware, microcode, hardware description languages, or any combination thereof. A076333-1064 / 06906 PATENTcode segment or machine-executable instructions may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, attributes, or memory contents. Information, arguments, attributes, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0089] The actual software code or specialized control hardware used to implement these systems and methods is not limiting of the invention. Thus, the operation and behavior of the systems and methods were described without reference to the specific software code being understood that software and control hardware can be designed to implement the systems and methods based on the description herein.

[0090] When implemented in software, the functions may be stored as one or more instructions or code on a non-transitory computer-readable or processor-readable storage medium. The steps of a method or algorithm disclosed herein may be embodied in a processor-executable software module which may reside on a computer-readable or processor-readable storage medium. A non-transitory computer-readable or processor-readable media includes both computer storage media and tangible storage media that facilitate transfer of a computer program from one place to another. A non-transitory processor-readable storage media may be any available media that may be accessed by a computer. By way of example, and not limitation, such non-transitory processor-readable media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other tangible storage medium that may be used to store desired program code in the form of instructions or data structures and that may be accessed by a computer or processor. Disk and disc, as used herein, include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media. Additionally, the operations of a method or algorithm may reside as one or any combination or set of codes and / or instructions on a non-transitory processor-readable medium and / or computer-readable medium, which may be incorporated into a computer program product.076333-1064 / 06906 PATENT

[0091] The preceding description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the following claims and the principles and novel features disclosed herein.

[0092] While various aspects and embodiments have been disclosed, other aspects and embodiments are contemplated. The various aspects and embodiments disclosed are for purposes of illustration and are not intended to be limiting, with the true scope and spirit being indicated by the following claims.

Claims

076333-1064 / 06906 PATENTCLAIMSWhat is claimed is:

1. A processor-implemented method for implementing improved Large Language Model (LLM) prompts using sensor data, the method comprising:obtaining, by at least one processor, sensor data generated by one or more sensors including a plurality of sensor records for a user, the plurality of sensor records including a plurality of subsets of the sensor data classified into a plurality of sensor data categories;receiving, by the at least one processor, a user-entered prompt via a user interface of a user device having the at least one processor;for each sensor data category, determining, by the at least one processor, a severity metric for the sensor data category based upon the sensor data of each sensor record of the subset of sensor records of the sensor data category;generating, by the at least one processor, a sensor aggregation file containing each subset of sensor data records and each severity metric for each category;instructing, by the at least one processor, a first call to an LLM in a first inference operation to generate a refined prompt based upon the user-entered prompt and the sensor aggregation file using the LLM;instructing, by the at least one processor, a second call to the LLM in a second inference operation to generate a context-aware response based upon the refined prompt and the sensor aggregation file; andgenerating, by the at least one processor, a user interface output including the context-aware response for display at the user interface of the user device.

2. The method of claim 1, wherein the LLM is installed on the user device and executed by the at least one processor the user device.

3. The method of claim 1, wherein the LLM is installed on a remote computing device that is remote to the user device having the at least one processor.

4. The method of claim 3, further comprising:transmitting, by the at least one processor, the user-entered prompt and the sensor aggregation file to the remote computing device via one or more networks; and076333-1064 / 06906 PATENTreceiving, by the at least one processor, the refined prompt from the remote computing device via the one or more networks.

5. The method of claim 3, further comprising:transmitting, by the at least one processor, the refined prompt and the sensor aggregation file to the remote computing device via one or more networks; andreceiving, by the at least one processor, the refined prompt from the remote computing device via the one or more networks.

6. The method of claim 1, further comprising determining, by the at least one processor, the sensor data category of a sensor data record based upon a preconfigured mapping between one or more types of sensor data of the sensor data record and the sensor data category.

7. The method of claim 1, wherein determining the severity metric for the sensor data category includes generating, by the at least one processor, the severity metric based upon a current sensor data record and each sensor data record of the subset of sensor data records having a timestamp satisfying a preconfigured timeframe interval.

8. The method of claim 7, further comprising:determining, by the at least one processor, one or more severity level thresholds for the sensor data category; anddetermining, by the at least one processor, a severity level of the sensor data category based upon comparing the severity metric against the one or more severity level thresholds.

9. The method of claim 1, further comprising determining, by the at least one processor, a query refinement goal for the sensor data category based upon the severity metric for the sensor data category, wherein the at least one processor obtains the refined prompt further based upon the query refinement goal.

10. The method of claim 1, wherein the sensor data categories of the sensor data include at least one of activity level, stress level, or sleep quality.076333-1064 / 06906 PATENT11. A system for implementing improved Large Language Model (LLM) prompts using sensor data, the system comprising:a user device comprising at least one processor configured to:obtain sensor data generated by one or more sensors including a plurality of sensor records for a user, the plurality of sensor records including a plurality of subsets of the sensor data classified into a plurality of sensor data categories;receive a user-entered prompt via a user interface of the user device having the at least one processor;for each sensor data category, determine a severity metric for the sensor data category based upon the sensor data of each sensor record of the subset of sensor records of the sensor data category;generate a sensor aggregation file containing each subset of sensor data records and each severity metric for each sensor data category;instruct a first call to an LLM in a first inference operation to generate a refined prompt based upon the user-entered prompt and the sensor aggregation file using the LLM;instruct a second call to the LLM in a second inference operation to generate a context-aware response based upon the refined prompt and the sensor aggregation file; and generate a user interface output including the context-aware response for display at the user interface of the user device.

12. The system of claim 11, wherein the LLM is installed on the user device and executed by the at least one processor the user device.

13. The system of claim 11, wherein the LLM is installed on a remote computing device that is remote to the user device having the at least one processor.

14. The system of claim 13, wherein the at least one processor is further configured to:transmit the user-entered prompt and the sensor aggregation file to the remote computing device via one or more networks; andreceive the refined prompt from the remote computing device via the one or more networks.076333-1064 / 06906 PATENT15. The system of claim 13, wherein the at least one processor is further configured to:transmit the refined prompt and the sensor aggregation file to the remote computing device via one or more networks; andreceive the refined prompt from the remote computing device via the one or more networks.

16. The system of claim 11, wherein the at least one processor is further configured to determine the sensor data category of a sensor data record based upon a preconfigured mapping between one or more types of sensor data of the sensor data record and the sensor data category.

17. The system of claim 11, wherein when determining the severity metric for the sensor data category the at least one processor is further configured to generate the severity metric based upon a current sensor data record and each sensor data record of the subset of sensor data records having a timestamp satisfying a preconfigured timeframe interval.

18. The system of claim 17, wherein the at least one processor is further configured to:determine one or more severity level thresholds for the sensor data category; and determine a severity level of the sensor data category based upon comparing the severity metric against the one or more severity level thresholds.

19. The system of claim 11, wherein the at least one processor is further configured to determine a query refinement goal for the sensor data category based upon the severity metric for the sensor data category, and wherein the at least one processor obtains the refined prompt further based upon the query refinement goal.

20. The system of claim 11, wherein the sensor data categories of the sensor data include at least one of activity level, stress level, or sleep quality.

Images