Systems and methods for distributed calendars

Abstract

Systems and methods in accordance with an embodiment of the invention provide a method for ingesting data. The method includes receiving a set of inputs, detecting content from the input, detecting intent for the inputs, and generating data structures based on the content types and intent.

Description

Systems and Methods for Distributed CalendarsFIELD OF THE INVENTION

[0001] The present invention generally relates to smart calendars and, more specifically, to distributed smart calendars that are capable of being grouped to provide different interactive views of a common set of calendar data.BACKGROUND

[0002] Smart calendars are a category of consumer electronics devices that are typically characterized by displaying a digital calendar on a touch screen display via an Internet connected device. Calendar information can be digitally displayed on a smart calendar that is often wall mounted or located upon a counter top in a similar manner to that in which conventional paper calendars are located in prominent locations within homes (and other settings) to provide a visual representation of calendar information. Smart calendars are often characterized in that they integrate calendar information from multiple user accounts on multiple calendar services. In this way, a smart calendar can be a useful way for multiple people (e.g. a household) to display calendar information that is updated as each user updates a personal calendar on a calendar service.SUMMARY OF THE INVENTION

[0003] Systems and methods for distributed calendars in accordance with embodiments of the invention are illustrated. One embodiment includes a method for generating structured calendar data, comprising receiving a plurality of inputs of different input types, wherein the input types include at least audio data, image data, and text data, processing each of the plurality of inputs through one of a plurality of format converters, wherein each format converter is configured to convert inputs from a specific input type into a standardized format, detecting an intent for each input of the plurality of inputs based on the standardized format, determining, via an engine selection element, which of a plurality of specialized import engines to utilize for each converted input based on the detected intent, generating, via the determined specialized import engines, prompts for a language model based on the converted input and the detected intent, generating language model outputs from the language model based on the generated prompts,processing outputs from the language model outputs to generate structured calendar data, wherein the structured calendar data includes at least one of calendar events, task lists, recipes, and meal plans, and storing the structured calendar data in a distributed calendar system.

[0004] In a further embodiment, the method further includes steps for distributing content to each of a plurality of devices.

[0005] In still another embodiment, at least two of the plurality of devices are part of a smart calendar group.

[0006] In a still further embodiment, each of the plurality of devices includes a different set of presentation settings and each of the plurality of devices presents the structured calendar data according to its set of presentation settings.

[0007] In yet another embodiment, the set of presentation settings includes at least one selected from the group consisting of a brightness setting, a time zone, a display preference, a font size, a color theme, and user-specific settings.

[0008] In a yet further embodiment, the set of presentation settings for at least one device of the plurality of devices includes a set of filters, wherein the particular device filters display of the structured calendar data based on the set of filters.

[0009] In another embodiment, the plurality of devices are linked to a common calendar group.

[0010] In another additional embodiment, receiving the plurality of inputs includes placing each of the plurality of inputs in a queue and processing each input from the queue.

[0011] In a further additional embodiment, processing each of the plurality of inputs includes converting a format of the input to a different second format prior to converting the input in the different second format into the standardized format.

[0012] In another embodiment again, processing each of the plurality of inputs includes detecting content from the set of inputs, wherein detecting content includes determining a type of the content, wherein the type of the content is at least one selected from the group consisting of lists, recipes, events, meal plans, and notes.

[0013] In a further embodiment again, detecting the intent is based on the type of the content.

[0014] In still yet another embodiment, processing each of the plurality of inputs includes detecting detail data from the set of inputs, wherein detail data includes at least one selected from the group consisting of dates, event recurrence, location details, contact information, titles, ingredients, cooking instructions, and list structures.

[0015] In a still yet further embodiment, detecting the intent includes determining whether the input is to be used to create at least one selected from the group consisting of calendar events, task lists, recipes, and meal plans.

[0016] In still another additional embodiment, the plurality of specialized import engines includes at least one selected from the group consisting of an event importer, a recipe importer, a list importer, and a meal generator.

[0017] In a still further additional embodiment, the language model is a large language model (LLM).

[0018] In still another embodiment again, the outputs from the language model are in a JSON format.

[0019] In a still further embodiment again, the set of inputs includes a list of food and processing outputs from the language model outputs to generate structured calendar data includes generating a meal plan and a shopping list based on the set of inputs.

[0020] One embodiment includes a non-transitory machine readable medium containing program instructions that are executable by a set of one or more processors to perform a method comprising receiving a plurality of inputs of different input types, wherein the input types include at least audio data, image data, and text data, processing each of the plurality of inputs through one of a plurality of format converters, wherein each format converter is configured to convert inputs from a specific input type into a standardized format, detecting an intent for each input of the plurality of inputs based on the standardized format, determining, via an engine selection element, which of a plurality of specialized import engines to utilize for each converted input based on the detected intent, generating, via the determined specialized import engines, prompts for a language model based on the converted input and the detected intent, generating language model outputs from the language model based on the generated prompts, processing outputs from the language model outputs to generate structured calendar data, wherein thestructured calendar data includes at least one of calendar events, task lists, recipes, and meal plans, and storing the structured calendar data in a distributed calendar system.

[0021] In yet another additional embodiment, the method further includes steps for distributing content to each of a plurality of devices, wherein at least two of the plurality of devices are part of a smart calendar group, each of the plurality of devices includes a different set of presentation settings and each of the plurality of devices presents the structured calendar data according to its set of presentation settings, and the set of presentation settings includes at least one selected from the group consisting of a brightness setting, a time zone, a display preference, a font size, a color theme, and userspecific settings.

[0022] In a yet further additional embodiment, the set of inputs includes a list of food and processing outputs from the language model outputs to generate structured calendar data includes generating a meal plan and a shopping list based on the set of inputs.

[0023] Additional embodiments and features are set forth in part in the description that follows, and in part will become apparent to those skilled in the art upon examination of the specification or may be learned by the practice of the invention. A further understanding of the nature and advantages of the present invention may be realized by reference to the remaining portions of the specification and the drawings, which forms a part of this disclosure.BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The description and claims will be more fully understood with reference to the following figures and data graphs, which are presented as exemplary embodiments of the invention and should not be construed as a complete recitation of the scope of the invention.

[0025] Fig. 1 illustrates a smart calendar system including multiple smart calendar devices configured as one or more smart calendar device groups in accordance with an embodiment of the invention.

[0026] Figs. 2A and 2B illustrate embodiments of smart calendar devices in accordance with various embodiments of the invention.

[0027] Figs. 3A - 3F illustrate user interfaces on a user device enabling a user to link a smart calendar device with a user account within a smart calendar system in accordance with an embodiment of the invention.

[0028] Fig. 4 illustrates a user interface generated by a calendar application on a smart calendar device, where the user interface displays a notification that a smart calendar device has been linked with a user account, in accordance with an embodiment of the invention.

[0029] Fig. 5 conceptually illustrates a household including multiple smart calendar devices, some of which are linked in a smart calendar device group, in accordance with an embodiment of the invention.

[0030] Fig. 6 conceptually illustrates data maintained by a smart calendar system with respect to smart calendar device groups linked to a user account in accordance with an embodiment of the invention.

[0031] Fig. 7 conceptually illustrates an example of a process for importing calendar events.

[0032] Fig. 8 illustrates an example of a graphical user interface (GUI) for importing events in accordance with an embodiment of the invention.

[0033] Fig. 9 illustrates an example of a GUI for importing a list in accordance with an embodiment of the invention.

[0034] Fig. 10 illustrates an example of a GUI for importing recipes in accordance with an embodiment of the invention.

[0035] Fig. 11 illustrates an example of a GUI for planning meals in accordance with an embodiment of the invention.

[0036] Fig. 12 illustrates an example of a GUI for adding new elements in accordance with an embodiment of the invention.

[0037] Fig. 13 illustrates an example of a GUI for profiles in accordance with an embodiment of the invention.

[0038] Fig. 14 illustrates an example of a GUI for profile rules in accordance with an embodiment of the invention.

[0039] Fig. 15 illustrates an example of a GUI for syncing calendars in accordance with an embodiment of the invention.

[0040] Fig. 16 illustrates example GUIs for viewing calendars in accordance with an embodiment of the invention.

[0041] Fig. 17 illustrates an example of a GUI for viewing tasks in accordance with an embodiment of the invention.

[0042] Fig. 18 illustrates another example of a GUI for viewing tasks in accordance with an embodiment of the invention.

[0043] Fig. 19 illustrates an example of a GUI for rewards in accordance with an embodiment of the invention.

[0044] Fig. 20 illustrates an example of a GUI for redeeming rewards in accordance with an embodiment of the invention.

[0045] Fig. 21 illustrates an example of a GUI for planning meals in accordance with an embodiment of the invention.

[0046] Fig. 22 illustrates an example of a GUI for interacting with lists in accordance with an embodiment of the invention.

[0047] Fig. 23 illustrates an example of a GUI for viewing lists in accordance with an embodiment of the invention.

[0048] Fig. 24 illustrates an example of a GUI for photo albums in accordance with an embodiment of the invention.

[0049] Fig. 25 illustrates an example of an architecture for distributed calendar systems in accordance with an embodiment of the invention.DETAILED DESCRIPTION

[0050] Turning now to the drawings, smart calendar devices that are capable of being grouped to provide different interactive views of a common set of calendar data in accordance with various embodiments of the invention are illustrated. In many embodiments, the smart calendar devices are utilized within a smart calendar system to display calendar data. In the context of smart calendar systems in accordance with many embodiments of the invention, calendar data is not limited to events but can include (but is not limited to) information concerning events, chores, lists, and / or meals. In a number of embodiments, the smart calendar system includes a smart calendar server system thatis responsible for continuously aggregating calendar data associated with user accounts from a variety of calendar services and / or user devices. In certain embodiments, the calendar data associated with a user account can be used to generate user interfaces displayed on one or more smart calendar devices linked with the user accounts. When multiple smart calendar devices are linked in a smart calendar device group, then the manner in which the same set of calendar data is used to generate user interfaces on specific smart calendar devices within the smart calendar device group will depend upon a set of presentation settings associated with the linked smart calendar device that are maintained by the smart calendar server system. Smart calendar systems incorporating smart calendar devices in accordance with various embodiments of the invention are discussed further below.Smart Calendar Systems

[0051] A smart calendar system in accordance with an embodiment of the invention is illustrated in Fig. 1 . The smart calendar system 100 includes a smart calendar server system 102 that communicates via a network 103 with multiple smart calendar devices 104, 106, 108, 110 associated with a variety of user accounts maintained within a database 112 managed by the smart calendar server system. Although smart calendar devices 104, 106, 108, 110 are illustrated as display type devices, smart calendar devices in accordance with many embodiments of the invention include devices of various types, such as (but not limited to) mobile devices, dedicated displays, personal computers, etc. The smart calendar server system 102 aggregates data from a variety of calendar services 114 such as, but not limited to, the Google Calendar service provided by Google, Inc. of Mountain View, California, the Outlook calendar service provided by Microsoft Corporation, of Redmond, Washington, and the iCal calendar service provided by Apple, Inc. of Cupertino, California. In a number of embodiments, the calendar server system is able to obtain calendar data from the calendar services, which is then associated with specific user accounts (or device groups or User Profiles) within the database 112. In some embodiments, calendar server systems may use various methods to obtain calendar data, such as (but not limited to), users may authenticate to a calendar service, a unique calendar URL may be utilized to access calendar data, etc. Calendar data inaccordance with certain embodiments of the invention is associated with a User Profile and / or a particular Device Group (e.g., a device or set of linked devices) within the distributed calendar service, rather than (or in addition to) the specific user's account who initiated a "sync calendar" action.

[0052] As can readily be appreciated, smart calendar server systems in accordance with various embodiments of the invention can integrate with any of a variety of calendar services as appropriate to the requirements of specific applications.

[0053] In several embodiments, the information that is displayed by the smart calendar devices 104, 106, 108, 110 is determined by the smart calendar server system 112. When a smart calendar device receives a user input requesting the display of a specific user interface view, the smart calendar device in accordance with some embodiments of the invention sends a request to the smart calendar server system, which provides information that can be utilized to generate the requested user interface. In some embodiments, the information is provided as a data structure (e.g. JSON object) that can be utilized to render a user interface display within a user interface. In many embodiments, the smart calendar server system returns text files (e.g. ICS formatted data, JSON files, files containing HTML and / or javacript, etc.) that can be utilized to render the user interface via a web rendering engine such as (but not limited to) the Webkit rendering engine. The display of a user interface including calendar data on a smart calendar device is shown in the photograph in Fig. 2A. Components of a smart calendar device that can be utilized within a smart calendar system in accordance with an embodiment of the invention is shown in Fig. 2B. While the smart calendar devices described herein are referred to as “calendars”, smart calendars devices can be utilized to provide a variety of information including (but not limited to) calendar events, photo albums, lists, recipes, meal planning, and / or task lists. The specific information that can be displayed can be configured by users and / or be customized to the requirements of specific applications in accordance with various embodiments of the invention.

[0054] Referring again to Fig. 1 , the smart calendar server system 102 also communicates via the network (e.g. the Internet) with a variety of user devices such as (but not limited to) mobile phones, tablets, and personal computers that execute software applications that enable users to configure their smart tablet devices. In this way, userdevices can provide an alternative user interface via which users can configure their user accounts within the smart calendar system, associate smart calendar devices with their user accounts, configure their smart calendar systems and upload information to the smart calendar system for use in the generation of user interfaces on the user’s smart calendar devices. A sequence of user interfaces presented to a user linking a smart calendar device to a user account in accordance with an embodiment of the invention is illustrated in Figs. 3A - 3F. In many embodiments, when a user completes the process of linking a smart calendar device to a user account via a user device, the smart calendar server system generates a notification that is pushed to the smart calendar device, which displays the notification. Display of a notification that a smart calendar device has been linked with a user account in accordance with an embodiment of the invention is shown in Fig. 4.

[0055] As can readily be appreciated, the specific user interfaces presented to users via user devices and / or the specific actions that users can take via their user devices with respect to the smart calendar system are not limited to those described above with respect to Figs. 1 - 4. The specific user interfaces presented via user devices within a smart calendar system implemented in accordance with various embodiments of the invention is largely dependent upon the requirements of specific applications.

[0056] Referring again to Fig. 1 , it can be observed that multiple smart calendar devices 104, 106, 108 access the network 103 (e.g. the Internet) via a common wireless access network (e.g. a home WiFi network). In many embodiments, user devices 116 can communicate with the smart calendar server system 102 via the same local network or from a different access point (118).

[0057] In a number of embodiments, a number of smart calendar devices 106, 108 can be linked with a user account to create a smart calendar device group 120. When a user account has multiple smart calendar devices within a smart calendar group, each of the smart calendar devices can display user interfaces generated by the smart calendar server system from a common set of data associated with the user account with which the smart calendar device group is associated. In this way, a household can utilize multiple smart calendar devices (typically in different rooms) that provide the members of the household with different views of a common set of calendar information associatedwith the user account for the household. While it is possible to link smart calendar devices to a smart calendar group, a household may also wish to have one or more additional smart calendar devices (104) that are not part of the smart calendar device group, but are instead linked to a separate user account. A household 500 including multiple smart calendar devices 502, 504 that form a smart calendar device group and one or more additional smart calendar devices 506 in accordance with an embodiment of the invention is conceptually illustrated in Fig. 5.

[0058] The manner in which a smart calendar server system links multiple smart calendar devices with a user account within the database of user account data that it maintains is conceptually illustrated in Fig. 6. In the illustrated embodiment, the user account has a set of calendar data associated with the user account. Each smart calendar device has a set of associated presentation settings, which define the manner in which the calendar data is presented via the user interface of the specific smart calendar device. Presentation settings in accordance with certain embodiments of the invention can include (but are not limited to) a brightness setting, a time zone, a display preference, a font size, a color theme, user-specific settings (e.g., child mode), etc. In this way, the same set of source calendar data can be utilized to generate different views based upon the individual presentation settings of each smart calendar device.

[0059] FIG. 6 conceptually illustrates a data model used by a smart calendar system to represent two or more smart calendar devices before and after a linking operation. Prior to linking, each smart calendar device maintains an independent set of local data 602, including (but not limited to) a set of events, tasks, lists, reminders, and other user-generated content. Each device also maintains its own local presentation settings 604, including such device-specific parameters as brightness, screen layout preferences, color themes, notification behavior, and other display or usability settings. As illustrated in the “Before Linking” column of FIG. 6, these data sets reside in separate data structures associated with each independent device.

[0060] When a user performs a linking operation, the system generates a shared calendar group associated with the user account (or household account) performing the link. As represented in the “After Linking” portion of FIG. 6, the system maintains shared data 612 that includes shared calendar content aggregated across the linked devices620. In many embodiments, this shared data includes shared data objects representing events, tasks, lists, reminders, and other content that are accessible by each linked device and are synchronized by the smart calendar server system. The shared data may receive updates from any linked device, and such updates are propagated to the other linked devices via the smart calendar server.

[0061] While the shared data centralizes and synchronizes user-facing calendar content, the system maintains device-specific settings 622 separately for each linked device. As shown in FIG. 6, each device continues to store its own presentation settings and device preferences which remain unshared even after linking. Thus, although multiple devices present a consistent set of calendar information derived from the shared data layer, each device can render that information differently according to device-specific constraints or user-configured preferences (e.g., brightness, color scheme, child-friendly view options, display density, etc.).

[0062] In some embodiments, the linking operation includes the creation of a shared identifier (e.g., a household identifier). Shared identifiers in accordance with various embodiments of the invention references the shared data while maintaining pointers to device-specific configuration blocks. This can allow the system to support hybrid behaviors in which content data are shared, but configuration data remain local. This architecture provides a robust permissioning model in which the system may:(i) allow certain content types to be shared while leaving sensitive or devicespecific data unshared;(ii) allow each device to express preferences regarding which shared content types it displays; and(iii) support heterogeneous device types (e.g., wall-mounted displays, bedside devices, mobile applications) with different presentation requirements but a unified underlying calendar data set.

[0063] As illustrated in FIG. 6, once devices are linked through the shared calendar group, the system may maintain bidirectional synchronization of shared data objects. Updates performed on one device’s user interface, such as (but not limited to) adding an event, marking a task complete, or modifying a list item, are written to the shared data and thereafter made available to all linked devices. In contrast, updates to device-specificpresentation settings (e.g., brightness adjustments on a bedside calendar device) are stored only in that device’s associated configuration block and do not propagate to other devices within the linked group.

[0064] This architecture provides numerous advantages in environments where multiple smart calendar devices are deployed within a household or other shared environment. For example, a large wall-mounted calendar device in a common area may share its events, tasks, and lists with a smaller bedside device used by a child, while each device maintains its own brightness settings, “kid mode” user interface settings, or devicespecific access restrictions. Thus, the system supports heterogeneous device experiences while maintaining shared situational awareness across the household.

[0065] While smart calendar systems including various smart calendar device group combinations are described above with reference to Figs. 1 - 6, it should be readily appreciated that any of a variety of smart calendar device groupings can be maintained by smart calendar server systems as appropriate to the requirements of specific applications in accordance with various embodiments of the invention. Various processes for importing calendar data and configuring presentation settings for smart calendar devices in accordance with various embodiments of the invention are discussed further below.Importing Calendar Data

[0066] Systems and methods in accordance with a number of embodiments of the invention may ingest data of various types from various sources to maintain the distributed calendar data. In several embodiments, ingested data can be used to create various calendar elements, such as (but not limited to) calendar events, lists, recipes, meal plans, etc.

[0067] An example of a process for importing calendar events is conceptually illustrated in Fig. 7. Process 700 receives (705) a set of inputs. Inputs in accordance with certain embodiments of the invention are received from various sources, such as (but not limited to) cameras, microphones, sensors, email applications, text messaging applications, etc. In a number of embodiments, inputs can be of various types such as (but not limited to) multimedia data (e.g., movies, photographs, voice recordings, etc.),text data (e.g., emails, text messages, websites, uniform resource locators (URLs), notes, tab-delimited text, etc.), formatted data (e.g., portable document format (PDF), MICROSOFT WORD documents, MICROSOFT EXCEL spreadsheets, etc ), etc.

[0068] In numerous embodiments, multiple inputs may be received and a queue of the received inputs is maintained as each input is processed. In various embodiments, inputs may be pre-processed to prepare the inputs for an ingestion process. Preprocessing in accordance with various embodiments of the invention includes (but is not limited to) format conversions (e.g., speech to text, PDF to text, encoded video to image frames, etc.), data enhancement (e.g., image processing, audio processing, etc.), etc. In some embodiments, format conversions convert tables (e.g., from emails, spreadsheets, etc.) to tab-delimited strings that can be used in further processing.

[0069] Process 700 detects (710) content from the set of inputs. Content that may be detected in accordance with numerous embodiments of the invention includes various types of content, such as (but not limited to) lists, recipes, events, meal plans, notes, etc. In certain embodiments, detecting content can include identifying and interpreting text from multimedia data, formatted data, etc. Processes in accordance with numerous embodiments of the invention detect various detail data associated with the content from the set of inputs. In many embodiments, detail data can include (but is not limited to) event details (e.g., dates, event recurrence, location details, contact information, titles, ingredients, cooking instructions, list structures, etc. In a number of embodiments, multiple types of content may be detected from a single input or set of inputs.

[0070] Process 700 detects (712) intents for the set of inputs. In numerous embodiments, processes detect intents for the set of inputs based on the detected content. For example, processes in accordance with certain embodiments of the invention may detect dates in the inputs and determine that the outputs should include events, detect ingredients and instructions and determine that the outputs should include a recipe, etc. In various embodiments, intents are detected more explicitly based on default settings or user inputs. For example, a desired output type may be specified by a user when the set of inputs are provided or determined based on other information (e.g., inputs received via email be identified as events, images may default to notes, etc.).

[0071] Process 700 generates (715) outputs based on the detected content and intent. In various embodiments, processes generate prompts for models (e.g., large language model (LLM)), where the outputs of the LLMs (e.g., in a JSON format) can be used to generate outputs (e.g., distributed calendar data structures) for the detected content. In some embodiments, outputs of the LLMs can be processed to generate the outputs of the process. For example, processing outputs of the LLMs can include (but is not limited to) identifying (or classifying) elements of the outputs (e.g., ingredients and instructions from a recipe), reformatting the data (e.g., converting from JSON to a different format), etc.

[0072] Outputs in accordance with some embodiments of the invention can include (but are not limited to) calendar events, journal entries, lists, recipes, meal plans, notes, etc. In a number of embodiments, outputs may include various visualizations of distributed calendar data, as illustrated throughout this description. Visualizations in accordance with several embodiments of the invention are presented on a device different from the device on which an input is captured. Different output types may be stored using different data structures in accordance with many embodiments of the invention.

[0073] Processes in accordance with a number of embodiments of the invention may generate outputs based on additional information beyond ingested data. Examples of other information include, but are not limited to, historic calendar data, user settings / preferences, dietary restrictions / requirements, nutrition goals, number of portions to prepare, desired nights of the week, favorite meals, available ingredients, etc. In many embodiments, smart calendar devices may automatically add ingredients for generated meal plans to a list (e.g., a grocery list).

[0074] In a number of embodiments, adding ingredients to a list includes a process for combining ingredients in the list. Combining ingredients in a list can include identifying interchangeable ingredients to simplify a shopping list and / or to reduce costs. For example, different recipes may call for sea salt, iodized salt, and table salt. Processes in accordance with a variety of embodiments of the invention may combine the amounts and / or elements to require a single amount of sea salt.

[0075] Processes in accordance with various embodiments of the invention may generate a number of meal plans based on inputs and / or additional information. In certainembodiments, processes may generate meal plans for a week (or other time period). In various embodiments, meal plans may be generated on a schedule (e.g., weekly) based on user input and / or an automated schedule. For example, processes may execute every Sunday morning to schedule meals for the week. Schedules in accordance with some embodiments of the invention may indicate meals to be generated (or excluded). In certain embodiments, generating meal plans on a schedule may include adding ingredients from recipes for the generated meal plans to a shopping list.

[0076] In some embodiments, processes utilize separate models (or modules) for prompt engineering for each of the different content types (e.g., lists, recipes, events, meals, etc.). Distributed calendar data structures in accordance with numerous embodiments of the invention can include various data structures such as (but not limited to) Recurring Rule (RRULE), XML data structures, JSON data structures, etc.

[0077] Prompt generation in accordance with a number of embodiments of the invention includes processing the detected content to generate the prompt. In numerous embodiments, processing detected content can include (but is not limited to) date formatting (e.g., for different international formats), collecting context data related to a prompt, etc. Context data in accordance with various embodiments of the invention can include (but is not limited to) historic data (e.g., previous consumed meals, attended events, previously built lists, etc.) and / or other distributed calendar data (e.g., upcoming events, user preferences, system settings, etc.). For example, processes in accordance with several embodiments of the invention may use a combination of previously consumed meals, a list of items currently in the refrigerator, and a set of user preferences to generate a meal plan for a number of meals over the next week.

[0078] In various embodiments, prompt engineering can reduce hallucinations by using specific prompts to focus the outputs of the generated prompts. For example, asking an LLM to find the RRULE in an image of an event flyer could cause the LLM to hallucinate. Processes in accordance with some embodiments of the invention reduce hallucinations by asking the LLM very clear, literal questions about the input data (e.g. building a RRULE from an LLM response to a question like "Which days of the week does the event recurs on?").

[0079] While specific processes for importing calendar data are described above, any of a variety of processes can be utilized to import distributed calendar data as appropriate to the requirements of specific applications. In certain embodiments, steps may be executed or performed in any order or sequence not limited to the order and sequence shown and described. In a number of embodiments, some of the above steps may be executed or performed substantially simultaneously where appropriate or in parallel to reduce latency and processing times. In some embodiments, one or more of the above steps may be omitted.

[0080] An example of a graphical user interface (GUI) for importing events in accordance with an embodiment of the invention is illustrated in Fig. 8. Events in accordance with some embodiments of the invention may be imported based on various documents, such as (but not limited to) PDFs, emails, photographs, audio recordings, websites, etc.

[0081] While specific implementations of importing events have been described above with respect to Fig. 8, there are numerous interfaces for importing events that may be implemented as appropriate to the requirements of a given application.

[0082] An example of a GUI for importing a list in accordance with an embodiment of the invention is illustrated in Fig. 9. Lists in accordance with some embodiments of the invention may be imported based on various documents, such as (but not limited to) PDFs, emails, photographs, audio recordings, websites, etc.

[0083] While specific implementations of importing lists have been described above with respect to Fig. 9, there are numerous interfaces for importing lists that may be implemented as appropriate to the requirements of a given application.

[0084] An example of a GUI for importing recipes in accordance with an embodiment of the invention is illustrated in Fig. 10. Recipes in accordance with some embodiments of the invention may be imported based on various documents, such as (but not limited to) PDFs, emails, photographs, audio recordings, websites, etc.

[0085] While specific implementations of importing recipes have been described above with respect to Fig. 10, there are numerous interfaces for importing recipes that may be implemented as appropriate to the requirements of a given application.

[0086] An example of a GUI for planning meals in accordance with an embodiment of the invention is illustrated in Fig. 11. In numerous embodiments, calendar applications can generate a meal plan. Meal plans in accordance with a variety of embodiments of the invention may include meal suggestions and / or recipes for a period of time (e.g., a week). In various embodiments, meal plans can be generated based on user preferences and / or input, such as (but not limited to) dietary restrictions / requirements, nutrition goals, ingredients, etc. In many embodiments, smart calendar devices may automatically add ingredients for generated meal plans to a list (e.g., a grocery list).

[0087] While specific implementations of planning meals have been described above with respect to Fig. 11 , there are numerous interfaces for planning meals that may be implemented as appropriate to the requirements of a given application.

[0088] An example of a GUI for adding new elements in accordance with an embodiment of the invention is illustrated in Fig. 12. This example includes examples of adding a recipe, an event, and a list.

[0089] While specific implementations of adding new elements have been described above with respect to Fig. 12, there are numerous interfaces for adding new elements that may be implemented as appropriate to the requirements of a given application.

[0090] An example of a GUI for profiles in accordance with an embodiment of the invention is illustrated in Fig. 13. Profiles in accordance with a variety of embodiments of the invention can be set up for different users. In some embodiments, profiles can be associated with calendars, chores, rewards, lists, etc. Profiles in accordance with various embodiments of the invention may be associated with characteristics (e.g., name, color, pattern, shape, etc.) that may be displayed through various GUIs of a smart calendar device to identify elements (e.g., calendars, chores, rewards, etc.) associated with different profiles. In many embodiments, an element may be associated with one or more profiles.

[0091] While specific implementations of profiles have been described above with respect to Fig. 13, there are numerous interfaces for profiles that may be implemented as appropriate to the requirements of a given application.

[0092] An example of a GUI for profile rules in accordance with an embodiment of the invention is illustrated in Fig. 14. In certain embodiments, profile rules can be used to assign calendar events to different profiles.

[0093] While specific implementations of profile rules have been described above with respect to Fig. 14, there are numerous configurations of profile rules that may be implemented as appropriate to the requirements of a given application.

[0094] An example of a GUI for syncing calendars in accordance with an embodiment of the invention is illustrated in Fig. 15. In certain embodiments, smart calendar devices can sync with calendars from various sources, such as (but not limited to) internal calendars, 3rdparty calendars (e.g., from GOOGLE, APPLE, MICROSOFT, etc.). In some embodiments, syncing calendars can be a two-way sync, where updates to the calendar made in a 3rdparty calendar are updated in the smart calendar device, and vice versa.

[0095] While specific implementations of syncing calendars have been described above with respect to Fig. 15, there are numerous interfaces for syncing calendars that may be implemented as appropriate to the requirements of a given application.

[0096] Example GUIs for viewing calendars in accordance with an embodiment of the invention is illustrated in Fig. 16. In numerous embodiments, calendar views may include a unified view that includes events from one, some, or all of the calendars associated with a profile. Calendar views in accordance with numerous embodiments of the invention may include views of different time ranges (e.g., daily, weekly, monthly, etc.). Smart calendar devices in accordance with certain embodiments of the invention may provide notifications and / or reminders for events from calendars associated with a profile. In some embodiments, notifications and / or reminders may be provided at one or more hardware devices associated a profile.

[0097] While specific implementations of viewing calendars have been described above with respect to Fig. 16, there are numerous interfaces for viewing calendars that may be implemented as appropriate to the requirements of a given application.

[0098] An example of a GUI for viewing tasks in accordance with an embodiment of the invention is illustrated in Fig. 17. Tasks in accordance with many embodiments of the invention can include chores and / or routines. Chores in accordance with manyembodiments of the invention can include due dates, recurrence, etc. In a variety of embodiments, routines can include tasks that are to be done continuously (e.g., daily, weekly, etc.). Routines in accordance with a variety of embodiments of the invention may be associated with a time of day (e.g., morning, afternoon, evening, a specific time, etc.). Tasks in accordance with a number of embodiments of the invention may include one or more of a title, due date, due time, recurring frequency, tokens (for rewards), associated profiles, etc.

[0099] Another GUI for viewing tasks in accordance with an embodiment of the invention is illustrated in Fig. 18.

[0100] While specific implementations of viewing tasks have been described above with respect to Figs. 17-18, there are numerous interfaces for viewing tasks that may be implemented as appropriate to the requirements of a given application.

[0101] An example of a GUI for rewards in accordance with an embodiment of the invention is illustrated in Fig. 19. In a number of embodiments, smart calendar devices allow users to assign tasks (or chores) and track the completion of those tasks to progress toward a reward. In exchange for completing tasks, processes in accordance with numerous embodiments of the invention may award tokens (e.g., stars) that may be saved and / or redeemed for rewards. Rewards (or tokens for rewards) in accordance with several embodiments of the invention may be of various types, including (but not limited to) one-time rewards, recurring rewards, time-limited rewards, etc. In many embodiments, smart calendar devices may provide an interface for viewing potential rewards as well as a cost (e.g., in tokens) required to redeem the reward.

[0102] An example of a GUI for redeeming rewards in accordance with an embodiment of the invention is illustrated in Fig. 20.

[0103] While specific implementations of rewards have been described above with respect to Figs. 19-20, there are numerous configurations for rewards that may be implemented as appropriate to the requirements of a given application.

[0104] An example of a GUI for planning meals in accordance with an embodiment of the invention is illustrated in Fig. 21.

[0105] While specific implementations of planning meals have been described above with respect to Fig. 21 , there are numerous interfaces for planning meals that may be implemented as appropriate to the requirements of a given application.

[0106] An example of a GUI for interacting with lists in accordance with an embodiment of the invention is illustrated in Fig. 22. Lists in accordance with a number of embodiments of the invention can include various items. In some embodiments, items of a list may be marked (e.g., via strikethrough) with a change in status (e.g., completed).

[0107] An example of a GUI for viewing lists in accordance with an embodiment of the invention is illustrated in Fig. 23.

[0108] While specific implementations of interacting with lists have been described above with respect to Figs. 22-23, there are numerous interfaces for interacting with lists that may be implemented as appropriate to the requirements of a given application.

[0109] An example of a GUI for photo albums in accordance with an embodiment of the invention is illustrated in Fig. 24. Photo albums in accordance with a variety of embodiments of the invention may include photos, videos, and / or audio. In a number of embodiments, content from photo albums may be used to generate a digital slideshow that may be displayed on hardware devices.

[0110] While specific implementations of photo albums have been described above with respect to Fig. 24, there are numerous interfaces for photo albums that may be implemented as appropriate to the requirements of a given application.Smart Calendar Device Hardware

[0111] As noted above, smart calendar devices and user devices in accordance with an embodiment of the invention are illustrated in Figure 2B. Smart calendar devices (and user devices) 200 in accordance with many embodiments of the invention can include (but are not limited to) one or more processors 205, one or more peripherals 210, one or more network interfaces 215, and memory 220. One skilled in the art will recognize that a distributed calendar element may exclude certain components and / or include other components that are omitted for brevity without departing from this invention.

[0112] The processor 205 can include (but is not limited to) a processor, microprocessor, controller, or a combination of processors, microprocessor, and / orcontrollers that performs instructions stored in the memory 220 to manipulate data stored in the memory. Processor instructions can configure the processor 205 to perform processes in accordance with certain embodiments of the invention. In various embodiments, processor instructions can be stored on a non-transitory machine readable medium.

[0113] Peripherals 210 can include any of a variety of components for capturing data, such as (but not limited to) cameras, displays (e.g., touch-screen), microphones, and / or sensors. In a variety of embodiments, peripherals can be used to gather inputs and / or provide outputs. The smart calendar device 200 can utilize network interface 215 to transmit and receive data over a network based upon the instructions performed by processor 205. Peripherals and / or network interfaces in accordance with many embodiments of the invention can be used to gather inputs that can be used to interact with distributed calendars.

[0114] Memory 220 can include a calendar application 225, and calendar data 230 utilized to generate user interfaces for display via a peripheral 210 of the smart calendar device. Calendar applications in accordance with several embodiments of the invention can be used to generate user interfaces that enable user interaction between the user and the smart calendar device and / or smart calendar system.

[0115] Although a specific example of a distributed calendar element is illustrated in this figure, any of a variety of distributed calendar elements can be utilized to perform processes for implementing distributed calendars similar to those described herein as appropriate to the requirements of specific applications in accordance with embodiments of the invention.

[0116] An example of an architecture for distributed calendar systems in accordance with an embodiment of the invention is illustrated in Fig. 25. Architecture 2500 illustrates format converter 2505, message handler 2510, engine selection element 2515, automatic speech recognition (ASR) model 2520, import engines 2525, language model 2530, and result processors (or persisters) 2535.

[0117] Format converters in accordance with some embodiments of the invention convert inputs between different formats. In this example, format converter 2505 is an optical character recognition (OCR) to tab-separated value (TSV) element. In a numberof embodiments, format converters can be specified to convert inputs for use with a particular language model. For example, distributed calendar systems in accordance with several embodiments of the invention convert text (e.g., OCR text) to tab-separated values as some large language models handle TSV’s quite gracefully. Format converters in accordance with a variety of embodiments of the invention pass converted outputs to a message handler.

[0118] Message handlers (e.g., PaulRevere::MessageHandler) in accordance with some embodiments of the invention ingest from a queue of upload messages. For example, when a photo is uploaded in a distributed calendar system in accordance with a number of embodiments of the invention, message is added to a message queue (e.g., by a skylight-fleet-typescript S3 bucket). In many embodiments, when an email pipeline receives a portable document format (PDF), it adds a message to a message queue. Message handlers can then ingest those messages and call an engine selection element.

[0119] Engine selection elements in accordance with many embodiments of the invention are responsible for deciding which engine to use for a particular input and calling that engine. In a number of embodiments, engine selection elements determine which engine to use based on the type of detected content from the inputs.

[0120] In numerous embodiments, engine selection elements (or distributed calendar systems) utilize ASR Models to process audio inputs prior to processing by import engines. For example, if an input file is an audio recording, processes in accordance with various embodiments of the invention may pass the audio through an automatic speech recognition model (e.g., OpenAI's Whisper) to convert it to text that may be processed by an import engine.

[0121] Import engines in accordance with several embodiments of the invention generate prompts that can be used by language models to generate outputs. In various embodiments, import engines structure function call parameters to generate prompts. Import engines in accordance with certain embodiments of the invention can perform various processes, such as (but not limited to) managing international data formatting, providing context about prior recipes or list data, etc. In a variety of embodiments, the prompt generation can be tailored with detail and context to help reduce hallucination.Language models in accordance with a number of embodiments of the invention may include a multimodal large language model (LLM) (e.g., OpenAI's GPT4o).

[0122] Result processors in accordance with many embodiments of the invention are responsible for translating language model outputs into the distributed calendar system datamodel. For example, asking a language model to find a recurring rule (RRULE) in an image of an event flyer could cause the LLM to hallucinate. Processes in accordance with several embodiments of the invention can significantly reduce hallucinations by asking the language model very clear, literal questions about the input data. Result processors in accordance with some embodiments of the invention format the outputs into the data model (e.g., building an RRULE from a language model response to a question like "Which days of the week does the event recurs on?").

[0123] Although a specific example of a distributed calendar system architecture is illustrated in this figure, any of a variety of distributed calendar system architectures can be utilized to perform processes for implementing distributed calendars similar to those described herein as appropriate to the requirements of specific applications in accordance with embodiments of the invention.

[0124] Although specific methods of distributed calendars are discussed above, many different methods of distributed calendars can be implemented in accordance with many different embodiments of the invention. It is therefore to be understood that the present invention may be practiced in ways other than specifically described, without departing from the scope and spirit of the present invention. Thus, embodiments of the present invention should be considered in all respects as illustrative and not restrictive. Accordingly, the scope of the invention should be determined not by the embodiments illustrated, but by the appended claims and their equivalents.

Claims

WHAT IS CLAIMED IS:1 . A method for generating structured calendar data, comprising: receiving a plurality of inputs of different input types, wherein the input types include at least audio data, image data, and text data; processing each of the plurality of inputs through one of a plurality of format converters, wherein each format converter is configured to convert inputs from a specific input type into a standardized format; detecting an intent for each input of the plurality of inputs based on the standardized format; determining, via an engine selection element, which of a plurality of specialized import engines to utilize for each converted input based on the detected intent; generating, via the determined specialized import engines, prompts for a language model based on the converted input and the detected intent; generating language model outputs from the language model based on the generated prompts; processing outputs from the language model outputs to generate structured calendar data, wherein the structured calendar data includes at least one of calendar events, task lists, recipes, and meal plans; and storing the structured calendar data in a distributed calendar system.

2. The method of claim 1 further comprising distributing content to each of a plurality of devices.

3. The method of claim 2, wherein at least two of the plurality of devices are part of a smart calendar group.

4. The method of claim 2 wherein each of the plurality of devices comprises a different set of presentation settings and each of the plurality of devices presents the structured calendar data according to its set of presentation settings.

5. The method of claim 4, wherein the set of presentation settings comprises at least one selected from the group consisting of a brightness setting, a time zone, a display preference, a font size, a color theme, and user-specific settings.

6. The method of claim 4, wherein the set of presentation settings for at least one device of the plurality of devices comprises a set of filters, wherein the particular device filters display of the structured calendar data based on the set of filters.

7. The method of claim 2, the plurality of devices are linked to a common calendar group.

8. The method of claim 1 , wherein receiving the plurality of inputs comprises placing each of the plurality of inputs in a queue and processing each input from the queue.

9. The method of claim 1 , wherein processing each of the plurality of inputs comprises converting a format of the input to a different second format prior to converting the input in the different second format into the standardized format.

10. The method of claim 1 , wherein processing each of the plurality of inputs comprises detecting content from the set of inputs, wherein detecting content comprises determining a type of the content, wherein the type of the content is at least one selected from the group consisting of lists, recipes, events, meal plans, and notes.

11. The method of claim 10, wherein detecting the intent is based on the type of the content.

12. The method of claim 1 , wherein processing each of the plurality of inputs comprises detecting detail data from the set of inputs, wherein detail data comprises at least one selected from the group consisting of dates, event recurrence, location details, contact information, titles, ingredients, cooking instructions, and list structures.

13. The method of claim 1 , wherein detecting the intent comprises determining whether the input is to be used to create at least one selected from the group consisting of calendar events, task lists, recipes, and meal plans.

14. The method of claim 1 , wherein the plurality of specialized import engines comprises at least one selected from the group consisting of an event importer, a recipe importer, a list importer, and a meal generator.

15. The method of claim 1 , wherein the language model is a large language model (LLM).

16. The method of claim 1 , wherein the outputs from the language model are in a JSON format.

17. The method of claim 1 , wherein the set of inputs comprises a list of food and processing outputs from the language model outputs to generate structured calendar data comprises generating a meal plan and a shopping list based on the set of inputs.

18. A non-transitory machine readable medium containing program instructions that are executable by a set of one or more processors to perform a method comprising: receiving a plurality of inputs of different input types, wherein the input types include at least audio data, image data, and text data; processing each of the plurality of inputs through one of a plurality of format converters, wherein each format converter is configured to convert inputs from a specific input type into a standardized format; detecting an intent for each input of the plurality of inputs based on the standardized format; determining, via an engine selection element, which of a plurality of specialized import engines to utilize for each converted input based on the detected intent; generating, via the determined specialized import engines, prompts for a language model based on the converted input and the detected intent;generating language model outputs from the language model based on the generated prompts; processing outputs from the language model outputs to generate structured calendar data, wherein the structured calendar data includes at least one of calendar events, task lists, recipes, and meal plans; and storing the structured calendar data in a distributed calendar system.

19. The non-transitory machine readable medium of claim 18 further comprising distributing content to each of a plurality of devices, wherein: at least two of the plurality of devices are part of a smart calendar group; each of the plurality of devices comprises a different set of presentation settings and each of the plurality of devices presents the structured calendar data according to its set of presentation settings; and the set of presentation settings comprises at least one selected from the group consisting of a brightness setting, a time zone, a display preference, a font size, a color theme, and user-specific settings.

20. The non-transitory machine readable medium of claim 18, wherein the set of inputs comprises a list of food and processing outputs from the language model outputs to generate structured calendar data comprises generating a meal plan and a shopping list based on the set of inputs.

Images