Ai-based digital family legacy platform for integrated management of digital tombstones, family relationships, memories, and legacy

Abstract

The present invention, on the basis of a digital tombstone provided in a cemetery, provides information and content of the deceased through display, input, power source, detection and communication functions and enables visitor recognition, audio and video output, energy self-sustained operation and remote management. In addition, artificial intelligence is used to automatically generate and supplement family trees, genealogies and photo / video-based family relationship data, and memory-associated spaces and an avatar reflecting family characteristics are provided in a metaverse space so that a digital family legacy is continuously managed. Furthermore, emotion inputs and interactions are analyzed using an ethical judgement model so as to refine and block risky expressions, and a safe and reliable digital posthumous digital persona curation service can be provided.

Description

A digital family heritage platform that integrates the management of AI-based digital tombstones, family relationships, memories, and legacy.

[0001] The invention according to the embodiment of the present disclosure is an invention relating to a service for the integrated management of family, ancestor, and memory assets based on digital technology and artificial intelligence.

[0002] A gravestone (also called a tombstone) is a symbolic structure installed at a grave to express memory and respect for the deceased. Gravestones are typically inscribed with the deceased's name, date of birth, date of death, and often a short epitaph.

[0003] As the name suggests, these gravestones are primarily made of stone materials such as marble or granite, and since the content, including inscriptions, is carved, there was a problem in that it was impossible to modify or add information and could not adapt to environmental changes.

[0004] To address these issues, gravestone technology utilizing digital electronic displays has recently been advancing, but it faces challenges in practicality due to energy management and environmental durability issues.

[0005] Furthermore, existing family tree generation tools largely rely on manual input, which presents limitations such as low user convenience and difficulties in integrated management with large-scale data. These systems can only function if accurate data is input by the user, and if information on family members is missing or incomplete, the reliability and usability of the generated family tree are diminished. In particular, historical genealogical documents, photos, and videos are not digitized, resulting in a lack of compatibility with modern technology. Traditional methods of family information management have limited capabilities for analyzing or integrally linking such data, making it difficult to guarantee data connectivity and completeness.

[0006] There is a growing need for the technological development of an innovative system that overcomes these limitations by utilizing not only user-inputted information but also AI and machine learning technologies to complement and sophisticatedly connect family relationships.

[0007] Furthermore, the metaverse environment has recently been evolving beyond simple visual experiences into a platform that reflects user identity and emotional experiences. Existing metaverse platforms primarily focus on specific functions such as gaming, shopping, and work; even when utilizing user data, they tend to concentrate solely on visual elements, failing to provide emotional connection or meaningful interaction. There is still a shortage of platforms that leverage family relationship data and memories to offer immersive emotional experiences encompassing the past, present, and future.

[0008] There is a need for the development of technology focused on overcoming these limitations to create realistic avatars based on family member data (e.g., appearance, voice, personality, memories, etc.) and to recreate places of memories in three dimensions (3D) to strengthen emotional connections between users.

[0009] The technical problem that the present invention aims to solve is, another objective of the present invention, to provide a technology that helps to remotely check the condition of a cemetery located in a remote and mountainous area where frequent visits are difficult due to its characteristics.

[0010] In addition, the purpose is to provide a family tree service device and a method for operating the device, which collect various information about family members from a user, analyze it to automatically generate and manage a family tree, and, in particular, include genealogical digitization, AI-based photo restoration, external social data integration, and automatic update functions to enhance user convenience.

[0011] Furthermore, the technical objective of the present invention is to provide a metaverse service device and a method for operating the device, which apply a metaverse platform based on family data collected from a user to a system where a user and family members can emotionally interact, thereby expanding family-centered data utilization through a security and consent system to enable the user to emotionally connect with the person they wish to see.

[0012] To achieve the aforementioned purpose, a digital tombstone according to an embodiment of the present invention, which is a tombstone installed in a cemetery and displays information including information or achievements of the deceased, comprises: a display unit that displays the operating status of the tombstone, characters, graphics, photos, and videos according to a command from a control unit; an input unit that inputs operation commands or data of the tombstone desired by a user and transmits the input information to the control unit; a storage unit that stores a basic program required for the operation of the cemetery and stores operation commands and data input through the input unit; a control unit that controls the operation of each component; and a power supply unit that supplies power required for the operation of each component.

[0013] In this embodiment, the display unit may be an electronic paper display device.

[0014] In this embodiment, the input unit may be a touch-type input device coupled to the display unit.

[0015] In the present embodiment, the power supply unit may include: a power generation means for generating electricity by at least one of wind power, solar thermal, and solar photovoltaic means; a charging means for storing electricity generated by the power generation means; and a power management means for managing a power amount including the amount of power generated through the power generation means, the amount of charge of the charging means, and the amount of power consumed by the monument.

[0016] In this embodiment, the power supply unit may include a power supply means that receives power from an external power supply means.

[0017] In this embodiment, a communication unit may be additionally included that is wired or wirelessly connected to a network providing a wired or wireless communication path, receives operation commands or data for a monument desired by a user, and transmits information including the operation status of the monument to the outside.

[0018] In this embodiment, the communication unit receives weather information from a network and transmits it to the control unit, and the control unit may additionally perform the operation of requesting the power supply unit to adjust the charging amount of the charging means based on the weather information.

[0019] In this embodiment, the detection unit further includes a detection unit that generates a human body detection signal and transmits it to the control unit when it detects a human body approaching within a predetermined distance or less around the monument; and the control unit may command the display unit to operate when the human body detection signal is for a predetermined time or longer.

[0020] In this embodiment, the control unit may additionally perform an operation of controlling the transmission of the human body detection signal to the outside through the communication unit.

[0021] In this embodiment, a sound unit that plays sound according to a sound playback command of the control unit may be additionally included.

[0022] In this embodiment, a shooting unit that captures a still image or a video according to a shooting command of the control unit may be additionally included.

[0023] In this embodiment, schedule information is managed, and a schedule unit is additionally included that generates schedule arrival information and transmits it to the control unit when a schedule entered through the input unit or communication unit arrives, and the control unit may additionally perform an operation of transmitting the schedule arrival information to the power management means when it receives the schedule arrival information from the schedule unit.

[0024] A family tree service device according to an embodiment of the present invention includes a communication interface unit that communicates with a user terminal device that uses a family tree service to form and manage relationships with family members, and a control unit that automatically generates a family tree based on family member information and family relationship data such as photos or videos provided by the user terminal device for generating the family tree, and automatically generates the family tree by applying artificial intelligence (AI) to supplement the missing information or data when the family member information or family relationship data is insufficient during the automatic generation of the family tree.

[0025] The above control unit can use the artificial intelligence, which has been trained on the photo or video-related learning data, to restore damaged parts in the photo or video provided as the family relationship data when automatically generating the family tree.

[0026] The control unit can automatically reflect the profiles, photos, and location data of the family members collected from an SNS server providing a Social Network Service (SNS) into the generated family tree after the family tree is created.

[0027] The above control unit can apply the artificial intelligence to extract text using OCR (Optical Character Recognition) when a photograph of a genealogy is provided, and can automatically reflect the content of the extracted text in the family tree.

[0028] The above control unit can perform a family record reinforcement operation to request the relatives of the family member whose data is missing to supplement the missing data if there is missing data among the family members.

[0029] A method for operating a family tree service device according to an embodiment of the present invention comprises: a communication interface unit communicating with a user terminal device that uses a family tree service to form and manage relationships with family members; and a control unit automatically generating a family tree based on family member information and family relationship data, such as photos or videos, provided by the user terminal device for generating the family tree, and, when the family member information or family relationship data is insufficient during the automatic generation of the family tree, applying artificial intelligence (AI) to supplement the insufficient information or data to automatically generate the family tree.

[0030] The step of automatically generating the family tree can be used when automatically generating the family tree by restoring damaged parts in the photos or videos provided as family relationship data by applying the artificial intelligence that has been trained on the photo or video-related learning data.

[0031] The step of automatically generating the family tree can automatically reflect the profiles, photos, and location data of the family members collected from the SNS server providing the SNS after the family tree is generated into the generated family tree.

[0032] The step of automatically generating the family tree above can apply the artificial intelligence to extract text using OCR when a photograph of the genealogy is provided, and automatically reflect the content of the extracted text in the family tree based on the content of the text.

[0033] The step of automatically generating the family tree above may perform a family record reinforcement operation that requests the relatives of the family member whose data is missing to supplement the missing data if there is missing data among the family members.

[0034] A metaverse service device according to an embodiment of the present invention includes a communication interface unit that communicates with a user terminal device of a user who intends to use a family relationship-centered metaverse service, and a control unit that applies family data provided by communication with the user terminal device to an artificial intelligence (AI) program to generate an avatar that reflects the characteristics of family members, reproduces a memory location related to the generated avatar as a virtual space, and provides a metaverse service to the user terminal device in which the generated avatar is active in the reproduced virtual space.

[0035] The control unit can analyze the family data by executing a deep learning model as the artificial intelligence program and apply the analysis results to a generative AI model to generate an avatar that reflects the characteristics of the family members.

[0036] The control unit analyzes data related to the appearance, voice, personality, and behavioral patterns of family members as family data to generate an avatar that reflects the characteristics of family members, and can implement the operation of the generated avatar through learning behavioral and conversational patterns using the artificial intelligence program.

[0037] The above control unit can analyze the family data to generate an avatar capable of emotional interaction that reflects changes in age between the past and present.

[0038] The control unit above can provide a service in the form of inviting a new family member to the metaverse service from the user terminal device, so that the family relationship-centered metaverse service is managed as a digital family legacy.

[0039] In addition, a method for operating a metaverse service device according to an embodiment of the present invention comprises the steps of: a communication interface unit communicating with a user terminal device of a user who intends to use a family relationship-centered metaverse service; and a control unit applying family data provided through communication with the user terminal device to an artificial intelligence (AI) program to generate an avatar that reflects the characteristics of a family member, reproducing a memory location related to the generated avatar as a virtual space, and providing a metaverse service in which the generated avatar is active in the reproduced virtual space to the user terminal device.

[0040] The step of generating the above avatar can be performed by running a deep learning model as the artificial intelligence program to analyze the family data, and applying the analysis results to a generative AI model to generate an avatar that reflects the characteristics of the family members.

[0041] The step of generating the above avatar involves generating an avatar that reflects the characteristics of a family member by analyzing data related to the family member's appearance, voice, personality, and behavioral patterns as the family data, and implementing the actions of the generated avatar through learning behavioral and conversational patterns using the artificial intelligence program.

[0042] The step of generating the above avatar can generate an avatar capable of emotional interaction that reflects past and present age changes by analyzing the family data.

[0043] The above driving method may further include a step in which the control unit provides a service in the form of inviting a new family member to the metaverse service from the user terminal device so that the family relationship-centered metaverse service is managed as a digital family legacy.

[0044] As stated above, the characteristic configuration of the present invention for achieving the objective of the present invention and realizing the characteristic effects of the present invention described below is as follows.

[0045] According to one aspect of the present invention, an ethical judgment device configured to analyze emotional input data of an agent, evaluate the risk level of said emotional input data, and refine or block said emotional input data when the risk level exceeds a preset standard, wherein the ethical judgment device acquires from a server emotional input data including at least some of the agent’s voice, handwriting, and text, and at least some of the agent’s non-verbal input data including at least some of the agent’s behavior, facial expression, gaze, posture, gesture, tone of voice, intonation, and breathing, and generates social ethics learning data composed of domestic and international codes of ethics, social norms, sensitive expressions specific to the culture to which at least some of said user and said agent belong, public institution standards, etc., and an ethical norm database storing said emotional learning data; generates social norm taboo expressions that violate the moral and ethical standards shared by the society to which said user belongs and social norm risk judgment criteria corresponding to said social norm taboo expressions based on said social ethics learning data, and community taboo expressions that violate the moral and ethical standards shared by the community to which said user and said agent belong and community norm risk judgment criteria corresponding to said community taboo expressions based on said emotional learning data. An ethical judgment device is disclosed, characterized by including a risk assessment processor that includes an ethical judgment model comprising at least some of a pre-trained artificial intelligence, a machine learning model, a deep learning network, and a statistical-based prediction model that generates a risk based on the social norm risk judgment criteria and the community norm risk judgment criteria.

[0046] For example, an ethical judgment device is disclosed, characterized in that the risk assessment processor further classifies agent sensitivity based on at least some of the agent's country, age group, cultural background information and relationship with the user, and further reflects the agent sensitivity in the risk level.

[0047] For example, an ethical judgment device characterized in that the risk assessment processor evaluates a real-time risk based on language response data including at least some of the operator's voice, handwriting, and text from an interaction with the operator, and emotional response data including at least some of the operator's behavior, facial expression, gaze, posture, gesture, tone of voice, intonation, and breathing, and adjusts the sensitivity in real-time based on the rate of change of the real-time risk.

[0048] For example, an ethical judgment device is disclosed, wherein the risk assessment processor further comprises a plurality of interaction learning models learned based on the interaction patterns of each of the user and the plurality of agents for each of the user and the plurality of agents corresponding to the emotion learning data, and generates at least some of the sensitivity and the risk based on the interaction learning responses of the plurality of interaction learning models, and the plurality of interaction learning models construct a response sequence based on at least some of the emotion learning data and the generated virtual scenarios, and generate the interaction learning response by repeatedly simulating the response sequence, and generates an interaction response suitability based on the risk and the sensitivity for the interaction learning response, and fine-tunes the interaction learning response based on the interaction learning response suitability.

[0049] For example, an ethical judgment device is disclosed, characterized in that the risk assessment processor acquires emotional response data, which is the response of the agent corresponding to emotional output data generated from a self-generation device, and includes a feedback loop structure that reinforces the interaction learning response having a low risk level and automatically corrects the interaction learning response having a high risk level based on the emotional response data and the interaction learning response.

[0050] For example, an ethics judgment device is disclosed, characterized by generating a weight for the operator sensitivity and dynamically adjusting the weight when the user status flag changes from alive to after death.

[0051] For example, an ethics judgment device is disclosed, characterized in that the risk assessment processor detects a risk level greater than a third standard risk level, which is a preset standard risk level within a preset time for the same agent based on the emotion learning data, and when the risk level greater than the third standard risk level exceeds a preset standard number of times, the processor determines the emotion learning data as emotion bias data and causes the server to delete the emotion bias data.

[0052] For example, an ethical judgment device is disclosed, characterized in that the ethical judgment model determines that if the risk level is determined to be greater than or equal to a preset first standard risk level or less than or equal to a preset second standard risk level which is greater than the first standard risk level, it determines it as a sensitive emotional expression and refines it into an appropriate expression, and if the risk level is determined to be greater than or equal to the second standard risk level, it determines it as a risk emotional expression and blocks the risk emotional expression.

[0053] According to one aspect of the present invention, a method of operating a digital self-curation system is disclosed, wherein the method comprises the steps of: a server acquiring from a receiving device emotional input data including at least some of a voice, handwritten text, and at least some of a person’s voice, handwriting, and text, and at least some of non-verbal input data including at least some of a person’s behavior, facial expression, gaze, posture, gesture, tone of voice, intonation, neural signal, biosignal, environmental information, and breathing; an ethical judgment device including an ethical judgment model including at least some of a pre-trained artificial intelligence, machine learning model, deep learning network, and statistical-based prediction model, calculating a risk level based on at least some of social ethical standards and community ethical standards for the emotional input data, and performing emotional refinement on the emotional input data when the risk level is judged to be greater than or equal to a preset standard risk level; and a self-generation device including a digital self-generation model including at least some of a pre-trained artificial intelligence, machine learning model, deep learning network, and statistical-based prediction model, generating emotional output data including at least some of corresponding verbal output data and non-verbal output data based on the emotional input data.

[0054] For example, a method of operation of a digital self-curation system is disclosed, wherein the server further stores self-generation basic data including at least some of a user’s conversation records, diaries, essays, memos, videos, audio, and handwritten notes; acquires the self-generation basic data from the self-generation device and the server; generates emotion learning data including at least some of the user’s textualized conversation records, text, and handwritten notes based on the self-generation basic data, and at least some of the user’s non-verbal learning data including at least some of the user’s behavior, facial expressions, gaze, posture, gestures, tone of voice, intonation, and breathing; and learns based on an emotion learning data pair including the emotion label data generated in correspondence with the emotion learning data through a pre-trained emotion inference model that generates label data in which emotions corresponding to the emotion learning data are labeled.

[0055] For example, the above-described ethical judgment device further comprises the steps of: extracting social taboo expressions including at least some of abusive language, verbal abuse, and discriminatory expressions regarding gender, race, origin, and religion defined by social norms, and generating a social norm risk level corresponding to said social taboo expressions; extracting community taboo expressions contrary to moral and ethical standards shared by the community to which said user and said agent belong based on said emotion learning data pairs, and generating a community norm risk level corresponding to said community taboo expressions; and generating a risk level based on said social norm risk level and said community norm risk level, wherein the ethical judgment model is characterized by being learned based on a learning data pair comprising a risk level input data including at least some of said social taboo expressions and said community taboo expressions generated from said self-generation basic data, and a risk level output data including at least some of said social norm risk level and said community norm risk level. A method of operation of a digital self-curation system is disclosed.

[0056] For example, a method of operation of a digital self-curation system is disclosed, characterized in that the ethical judgment device further performs at least some of the steps of: softening the expression of risk emotion when the risk level is determined to be greater than or equal to a preset first standard risk level or less than or equal to a preset second standard risk level greater than the first standard risk level; and blocking the expression of risk emotion when the risk level is determined to be greater than or equal to the second standard risk level.

[0057] For example, a method of operation of a digital self-curation system is disclosed, wherein the self-generating device further comprises the step of converting the emotion output data into reaction data corresponding to an emotion output device, and causing the emotion output device, which includes at least some of a digital monument, UI, display, hologram device, haptic device, speaker, robot, VR device, AR device, and mobile device, to output the reaction data, wherein the reaction data includes at least some of text, voice, avatar, light, image, hologram, multimodal signal, and robot motion.

[0058] For example, a method of operation of a digital self-curation system is disclosed, wherein the emotion output device comprises at least some of a digital monument, a UI, and a metaverse, and outputs the reaction data through at least some of text, voice, an avatar, light, an image, a hologram, a multimodal system, and a robot.

[0059] For example, the server obtains from the receiving device emotional response data including at least some of the voice, handwriting, and text of the operator corresponding to the emotional output data, and at least some of the non-verbal response data including at least some of the operator's behavior, facial expression, gaze, posture, gesture, tone of voice, intonation, and breathing; the self-generating device generates emotional labeling data corresponding to the emotional response data and generates a response learning data pair including the emotional response data and the emotional labeling data; and the server

[0060] A method of operation of a digital self-curation system is disclosed, characterized by storing the emotion output data and the emotion response data including generated time information, and further including the step of the ethics judgment device and the self-generation device performing self-correction based on the response learning data pair.

[0061] For example, a method of operation of a digital self-curation system is disclosed, characterized in that the server acquires relationship data including at least some of blood relationship data, friendship relationship data, and role relationship data between the agent and the user, and at least one of the self-generating device and the ethics judgment device further performs the steps of generating a relationship graph corresponding to the agent based on the relationship data, generating a relationship diagram based on at least some of the relationship graph and the rate of change of the relationship graph, and reflecting the relationship diagram in the emotion output data and the risk level.

[0062] For example, a method of operation of a digital self-curation system is disclosed, wherein the server further includes the step of activating a preemptive utterance trigger when a user status flag is changed from living to deceased, and the step of generating emotion output data when the preemptive utterance trigger is detected, wherein the preemptive utterance trigger includes at least one of an anniversary, a calendar event, a geofence, and a connection of an operator terminal.

[0063] According to the present invention, a tombstone capable of adapting to changes in the times and environment can be provided by integrating digital technology while maintaining the form of a traditional tombstone.

[0064] In addition, according to the present invention, a tombstone can be provided in which the content displayed on the tombstone can be freely modified.

[0065] In addition, according to the present invention, it is possible to provide a monument that can be installed independently by enabling efficient power generation and energy management even in situations where power infrastructure is poor, such as in remote mountainous areas.

[0066] In addition, according to the present invention, a gravestone can be provided that allows the bereaved family to check the condition by transmitting the situation of the area where the gravestone is installed to the outside using a sensing unit and a shooting unit.

[0067] In addition, according to the present invention, a monument capable of sharing data with an external device through a communication unit can be provided.

[0068] According to an embodiment of the present invention, a modern and accessible data environment is established through the digitization and restoration of old data, and large-scale data integration and continuous updates are enabled by automatically integrating external data through social platform and API (Application Programming Interface) linkage. This improves the user experience and presents a new paradigm that supports systematic family tree management and utilization.

[0069] Furthermore, embodiments of the present invention enable users to manage family information with minimal effort through AI-based automated data processing and visual tools (improving user convenience), and digitize traditional materials such as genealogies, photos, and videos to increase the possibility of intergenerational transmission and continuously pass them on to descendants (preservation of family heritage). In addition, through data refinement and supplementation using AI, family relationships can be connected and maintained based on accurate information (ensuring data accuracy and reliability), strengthening family bonds and emotional values, and contributing to the recording and preservation of family history in the digital age (creation of social value).

[0070] Furthermore, embodiments of the present invention provide a tool that systematically manages user data and enables an intuitive understanding of family relationships, and can create a new user experience by integrating modern technology with traditional family data.

[0071] According to an embodiment of the present invention, an immersive emotional experience encompassing the past, present, and future can be provided by utilizing family relationship data and memories.

[0072] In addition, embodiments of the present invention can strengthen emotional connections between users by generating realistic avatars based on data of family members and reproducing places of memories in 3D.

[0073] Furthermore, embodiments of the present invention can promote cultural understanding by sharing family memories or places with other users through a global memory exchange program, support seamless interaction with users of other languages ​​through translation and interpretation AI, and enable family avatars of various nationalities to interact in the metaverse on specific anniversaries (e.g., World Family Day) through global events and communities.

[0074] Embodiments of the present invention can expand the value of inspiration and connection globally by extending a family data-based platform into a global community.

[0075] The ethical judgment device analyzes emotional input data including the operator's verbal and non-verbal expressions, evaluates the risk level of the emotional input data by reflecting social and communal characteristics, and has the effect of refining or blocking the emotional input data based on a preset standard risk level.

[0076] It provides an advanced understanding of emotion through multi-channel analysis including verbal and non-verbal input data, and generates personalized emotional output for the user, thereby enabling emotional exchange similar to emotional interaction with the user.

[0077] FIG. 1 is a block diagram of a digital tombstone according to the present invention.

[0078] FIG. 2 is a flowchart showing the operation of the control unit in FIG. 1.

[0079] FIG. 3 illustrates the appearance of a digital tombstone according to the present invention.

[0080] FIG. 4 is a block diagram illustrating an environment in which a digital monument according to the present invention is used.

[0081] FIG. 5 is a drawing showing a family tree service system according to an embodiment of the present invention.

[0082] Figure 6 is a block diagram illustrating the detailed structure of the family tree service device of Figure 5.

[0083] Figure 7 is a block diagram illustrating the detailed structure of the family tree service section of Figure 6.

[0084] Figure 8 is a flowchart showing the operation process of the family tree service device of Figure 5.

[0085] FIG. 9 is a diagram showing a metaverse service system according to an embodiment of the present invention.

[0086] FIG. 10 is a block diagram illustrating the detailed structure of the metaverse service device of FIG. 9.

[0087] Figure 11 is a block diagram illustrating the detailed structure of the metaverse service section of Figure 10.

[0088] FIG. 12 is a flowchart illustrating a metaverse service process according to an embodiment of the present invention.

[0089] Figure 13 is a flowchart showing the operation process of the metaverse service device of Figure 9.

[0090] FIG. 14 schematically discloses a conceptual diagram exemplarily illustrating a digital self-curation system according to one embodiment of the present invention.

[0091] FIG. 15 schematically illustrates a conceptual diagram exemplifying the configuration of an ethics judgment device according to one embodiment of the present invention.

[0092] FIG. 16 schematically illustrates a flowchart showing the operation method of an ethics judgment device according to one embodiment of the present invention.

[0093] FIG. 17 schematically illustrates a conceptual diagram exemplifying the configuration of a self-generation device according to one embodiment of the present invention.

[0094] FIG. 18 schematically illustrates a flowchart showing the operation method of a digital self-curation system according to one embodiment of the present invention.

[0095] Hereinafter, digital monuments according to embodiments of the present invention will be described for each embodiment with reference to the attached drawings.

[0096] In the following description of the present invention, specific descriptions of related known functions or configurations will be omitted if it is determined that such descriptions would unnecessarily obscure the essence of the invention. Furthermore, the terms described below are established considering their functions in the present invention; these terms may vary depending on the intention or practice of the manufacturer producing the product, and their definitions should be based on the content throughout this specification.

[0097] A digital monument according to the present invention will be described below with reference to the attached drawings FIG. 1 and FIG. 2.

[0098] FIG. 1 is a block diagram of a digital monument according to the present invention, and FIG. 2 is a flowchart showing the operation of the control unit in FIG. 1.

[0099] As described above, the digital tombstone (1100) according to the present embodiment includes an input unit (1110), a display unit (1120), a communication unit (1130), a detection unit (1140), a control unit (1150), a sound unit (1160), a schedule unit (1170), a storage unit (1180), a shooting unit (1196), and a power supply unit (1200), and is installed in a cemetery, etc. to display information including information or achievements about the deceased.

[0100] The display unit (1120) displays the operating status of the monument, characters, shapes, photos, and videos according to the command of the control unit (1150). Here, the display unit (1120) is preferably an electronic paper display device in consideration of poor power infrastructure. As is well known, since electronic paper display devices use almost no power when the displayed content does not change, limited power resources can be utilized efficiently. However, the present invention is not particularly limited to this, and various well-known thin-film display devices may be used.

[0101] The input unit (1110) inputs operation commands or data for the digital monument (1100) desired by the user and transmits the input information to the control unit (1150). It is preferable that the input unit (1110) use a touch-type input device coupled to the display unit (1120). As is well known, a touch-type input device such as a touch screen can provide a superior input environment compared to a keyboard or keypad by enabling not only point touch but also multipoint touch and drawing. However, it should be noted that the input of the user's operation commands does not need to be input only through the input unit (1110), and can also be input from outside the digital monument (1100) through the communication unit (1130) to be described later.

[0102] The storage unit (1180) stores basic programs (firmware, OS) required for grave operation and stores operation commands and data input through the input unit (1110). The storage unit (1180) stores not only information input through the input unit (1110) but also information received through the communication unit (1130). Furthermore, considering low-power situations, it is desirable for the storage unit (1180) to be formed as a non-volatile memory in which the stored contents are not erased even when there is no power supply. Additionally, for semi-permanent storage, the stored contents of the storage unit (1180) may be backed up to a separate storage space via a network. In this case, data transmission for backup is preferably performed through batch processing rather than real-time, thereby enabling efficient power management by processing in batches when there is a surplus of power. Meanwhile, it is desirable for the storage unit (1180) to store audio files of the deceased's voice from their lifetime and photos of the deceased.

[0103] The power supply unit (1200) supplies power necessary for the operation of each component. The power supply unit (1200) preferably includes a power supply means that receives power from an external power supply means. However, the places where the tombstone is installed are generally mountainous areas or cemeteries where power infrastructure is not established. Therefore, relying solely on a power supply means inevitably imposes significant limitations on the installation location of the digital tombstone (1100). To address this, the power supply unit (1200) according to the present invention preferably includes a power generation means, a charging means, and a power management means.

[0104] The power generation means produces electricity using at least one of wind power, solar thermal, and solar photovoltaic power. The amount of electricity generated using the power generation means can be appropriately adjusted by considering the storage capacity of the charging means, the characteristics of the installation location, the visiting frequency and duration of visits by bereaved families, etc. In this case, if the power generation means is solar thermal or solar photovoltaic, it is desirable to maximize power generation efficiency by utilizing Maximum Power Point Tracking (MPPT) technology.

[0105] The charging means stores electricity generated by the power generation means. The charging means is a secondary battery capable of multiple charging and discharging cycles, and considering the usage environment of the gravestone, it is preferable to use a lithium iron phosphate (LiFePO4) battery. However, the present invention

[0106] The battery is not specifically limited to this and can be made of various well-known materials. Additionally, the battery capacity can be selected by considering the aforementioned power generation capacity, the power consumption of the gravestone, and the frequency or duration of visits by the bereaved family.

[0107] The power management means manages the total amount of power, including the amount of power generated through the power generation means, the amount of power charged by the charging means, and the amount of power consumed by the gravestone. For example, it is not necessarily good for the battery, which is the charging means, to be simply fully charged. As is well known, if there is no power usage while fully charged, battery degradation occurs more quickly; therefore, depending on the battery material, it may be desirable to charge it to about 20-50% for long-term standby or lifespan. However, if the amount of power is limited considering only battery lifespan, it may not be possible to meet the amount of power used during holidays such as Chuseok or Lunar New Year, or on the anniversary of a deceased person's death; therefore, the power management means can appropriately adjust the amount of power generated, the amount of power charged, and the amount of power discharged by considering the schedule of the schedule unit (1170) to be described later and weather information received through the communication unit (1130).

[0108] The communication unit (1130) is connected via wired or wireless means to a network that provides a wired or wireless communication path, receives operation commands or data for the monument desired by the user, and transmits information including the operation status of the monument to the outside. Since the location where the monument is installed often has poor communication infrastructure, it is preferable for the communication unit (1130) to use wireless communication using a mobile phone communication protocol. Additionally, it should be noted that, depending on the case, communication may be performed using communication protocols such as Bluetooth, Zigbee, or Wi-Fi to connect with a user terminal and transmit data to the outside.

[0109] The communication unit (1130) also performs the operation of receiving weather information from the network and transmitting it to the control unit. Upon receiving this, the control unit (1150) can additionally perform the operation of requesting the power supply unit to adjust the charging amount of the charging means based on the weather information.

[0110] To explain this in detail, if the aforementioned power generation means is wind power, the amount of power that can be generated can be predicted according to the wind speed included in the weather information. Therefore, the power management means can manage whether or not to fully charge the charging means by referring to the schedule information of the schedule part (1170). Likewise, it should be noted that even if the power generation means is solar thermal or solar photovoltaic, it is also possible to adjust the amount of charging according to the amount of clouds included in the weather information.

[0111] The detection unit (1140) generates a human body detection signal and transmits it to the control unit (1150) when it detects a human body approaching within a predetermined distance from the gravestone (1100). The human body detection detected through the detection unit (1140) can be usefully utilized to convey the situation around the grave to the bereaved family in conjunction with the camera unit (1196) to be described later. Additionally, the detection unit (1140) can help the control unit (1150) determine whether all functions of the gravestone (1100) will be activated when bereaved family members are present around the gravestone (1100). Meanwhile, to prevent malfunctions caused by the detection of wild animals or passersby, it is desirable for the detection unit (1140) to generate a human body detection signal only for detections exceeding a predetermined time and a predetermined size. If the human body detection signal exceeds a predetermined time, the control unit (1150) may command all components of the gravestone (1100), including the display unit (1120), to be activated. Additionally, the control unit (1150) may additionally perform an operation to control the transmission of a human body detection signal to the outside through the communication unit (1130).

[0112] Through this, all or part of the bereaved family members receive this, and receive a still image or video captured through the camera unit (1196) to check whether weeding is being performed around the gravestone, whether a memorial service is being performed, whether there is any damage, or whether there is a disaster situation such as a forest fire.

[0113] The sound unit (1160) plays a sound according to the sound playback command of the control unit (1150). The sound played may be a sound to scare away wild animals, and during the ritual, music that the deceased liked or background sound suitable for the ritual may be used.

[0114] The camera unit (1196) captures a still image or a video in accordance with the camera command of the control unit (1150). The still image or video captured by the camera unit (1196) may be used as a record of the memorial service for the bereaved family, and may also be used as a surveillance record for verifying disaster situations such as fire, or for preventing damage caused by wild animals.

[0115] The schedule unit (1170) manages schedule information, and when a schedule entered through the input unit (1110) or the communication unit (1130) arrives, it generates this as schedule arrival information and transmits it to the control unit (1150). As the digital gravestone (1100) according to the present invention is equipped with the schedule unit (1170), the power management efficiency of the power unit (1200) can be increased, and a more accurate situational judgment can be made regarding detection signals that are not on the schedule. That is, efficient operation of the gravestone can be determined based on whether it matches schedules such as grave cleaning days, in addition to general grave visiting days such as Chuseok, Seollal, Hansik, and Cheongmyeong, or specific days such as memorial days. When the control unit (150) receives the schedule arrival information from the schedule unit (1170), it additionally performs the operation of transmitting this to the power management means.

[0116] The control unit (1150) controls the operation of each component. Referring to the attached drawing Fig. 2, the operation of the digital tombstone (1100) including the control unit (1150) is described in more detail. First, information is received through the input unit (1110) or the communication unit (1130). The information received at this time may include the anniversary of the deceased, the scheduled date of visit by the bereaved family, and the selection of sound to be displayed on the display unit (1120) and output from the sound unit (1160).

[0117] The schedule unit (1170) checks whether the schedule corresponding to the schedule information entered in the aforementioned step has arrived, and if it has not arrived, the power is managed in a standby state, and if the schedule has arrived, the power is managed in a normal power state.

[0118] Subsequently, the system investigates whether there is an information update; if there is updated information, it updates and stores it in the storage unit (1180), and if there is no information, it proceeds to a step of investigating whether to terminate. Through this, the digital monument (1100) is able to automatically and manually maintain the system's information in an up-to-date state.

[0119] Meanwhile, it is desirable for the control unit (1150) to identify the visitor through the Bluetooth device paired with the Bluetooth of the communication unit (1130) and to output a message containing the visitor's name using the voice of the deceased stored in the storage unit according to the visitor.

[0120] Examples of use of the digital monument according to the present invention are described below with reference to the attached drawings.

[0121] FIG. 3 illustrates the appearance of a digital tombstone according to the present invention, and FIG. 4 is a block diagram illustrating the environment in which a digital tombstone according to the present invention is used.

[0122] As illustrated in FIG. 3, the digital monument (1100) according to the present example of use is depicted as being erected on the ground, but the present invention is not specifically limited thereto and may be in the form of a flat monument, and may also be applied to monuments, etc.

[0123] In addition, since the installation location of the monument is outdoors, it is desirable that all components, including the housing (1101), be protected by waterproof and dustproof standards so that they can withstand all weather conditions.

[0124] A solar panel (1201) is installed on the upper part of the monument as a power generation means constituting the power supply unit, and electricity is produced using sunlight.

[0125] A camera (1190) is installed on one side of the front of the monument as a component of a detection unit (1140) and a shooting unit to detect a human body and, as needed, capture a still image or video. The captured image information is transmitted to a user terminal (1210) via a communication unit (1130) as shown in FIG. 4, thereby helping the user check the surroundings of the monument.

[0126] In addition, the communication unit (1130) of the tombstone can transmit information captured by the camera (1190) to the cemetery management server (1230) based on the judgment of the control unit (1150) or the request of the user terminal (1210), so that necessary measures can be taken if a cemetery manager is present. If there is no separate cemetery management system, a cemetery management service provider operating a separate service server (1220) may dispatch necessary personnel to take necessary measures around the cemetery.

[0127] FIG. 5 is a drawing showing a family tree service system according to an embodiment of the present invention.

[0128] As illustrated in FIG. 5, a family tree service system (2090) according to an embodiment of the present invention is a system comprising a user data-based family tree automatic generation and management platform, and includes some or all of a user terminal device (2100), a communication network (2110), a family tree service device (2120), and a third-party device (2130).

[0129] Here, "including some or all" means that some components, such as a third-party device (2130), may be omitted to configure the family tree service system (2090), or that some or all of the components constituting the family tree service device (2120) may be integrated into a network device (e.g., gateway, wireless exchange device, etc.) constituting the communication network (2110). To facilitate a sufficient understanding of the invention, it is described as including all.

[0130] The user terminal device (2100) may include various types of terminal devices for accessing the family tree service device (2120) of FIG. 5 to use services such as automatic generation and management of a family tree based on user data. The user terminal device (2100) may include PC-based terminal devices such as desktop computers or laptop computers, and may also include mobile-based terminal devices such as smartphones, tablet PCs, and wearable devices worn by users on their wrists, etc. Through PC-based terminal devices, the family tree service can be used by utilizing web services, and through mobile-based terminal devices, the family tree service according to the embodiment of the present invention can be used by utilizing application (hereinafter, App.) services. Through mobile-based terminal devices, data for the automatic generation of a family tree can be provided through the app screen by simply running the app, and the corresponding screen can be checked, and furthermore, notifications can be received in the form of push messages. On the other hand, PC-based terminal devices may be used for additional services such as notifications, such as email, text messages (SMS), and messenger-based chat services.

[0131] The user of the user terminal device (2100) may, for instance, be a family member, and here, the family member may refer to a unit such as a clan or a lineage, but is not limited thereto; it is also possible to specify the scope of direct ancestors or direct descendants, and the user using the service may specify the scope of the family. Furthermore, using the user terminal device (2100), users may provide information such as surname, age, and location based on family member information, or provide materials (or data) such as photos or videos. Additionally, users may use the user terminal device (2100) to provide detailed information such as occupation, personality, voice, and memories. Moreover, users may use the user terminal device (2100) to photograph or scan traditional genealogical documents and provide them.

[0132] Furthermore, the user terminal device (2100) according to an embodiment of the present invention can access the service of the family tree service device (2120) of FIG. 5 and view a service screen of the family tree, i.e., a visualized graphic (GUI) screen, which is generated based on various types of information or data provided. In this process, the user of the user terminal device (2100) can easily explore the history of the family. For example, if an item such as a genealogy is selected on the screen, the user can view content related to their ancestor, intermediate ancestor, branch ancestor, etc., and in the case of direct ancestors or direct descendants, they can view recent activities within a designated range up to the third generation ancestor or grandson, and furthermore, by specifying the range of relatives, they can also view the recent activities of relatives such as fourth cousins ​​or eighth cousins.

[0133] The communication network (2110) can be configured in various forms. The communication network (2110) may include both wired and wireless communication networks. For example, a wired or wireless internet network may be used or interconnected as the communication network (2110). Here, the wired network includes internet networks such as cable networks or public telephone networks (PSTN), and the wireless communication network includes CDMA, WCDMA, GSM, EPC (Evolved Packet Core), LTE (Long Term Evolution), WiBro networks, and public or private wireless networks based on LPWA (Low Power Wide Area) used for the application of Internet of Things (IoT) technology recently. Of course, the communication network (2110) according to the embodiment of the present invention is not limited thereto and may be used as an access network for a next-generation mobile communication system to be implemented in the future, for example, a cloud computing network under a cloud computing environment, a 5G network, a 6G network, etc. For example, if the communication network (2110) is a wired communication network, the access point within the communication network can be connected to the exchange of a telephone company, etc., but if it is a wireless communication network, it can be connected to an SGSN or GGSN (Gateway GPRS Support Node) operated by a telecommunications company to process data, or connected to various relay stations such as a BTS (Base Transceiver Station), NodeB, e-NodeB to process data.

[0134] The communication network (2110) may include access points. Access points may include small base stations, such as femto or pico base stations, which are often installed within a building. Here, femto or pico base stations may be classified according to the maximum number of user terminal devices (2100) of FIG. 5 that can be connected. Of course, user terminal devices (2100) may include a short-range communication module for performing short-range communication such as Zigbee and Wi-Fi. Access points may use TCP / IP or RTSP (Real-Time Streaming Protocol) for wireless communication. Here, short-range communication may be performed using various standards other than Wi-Fi, such as Bluetooth, Zigbee, infrared (IrDA), RF (Radio Frequency) such as UHF (Ultra High Frequency) and VHF (Very High Frequency), and ultra-wideband communication (UWB). Accordingly, the access point can extract the location of the data packet, specify the best communication path for the extracted location, and transmit the data packet along the specified communication path to the next device, such as a family tree service device (2120). The access point may share multiple lines in a typical network environment and may include, for example, routers, repeaters, and relays.

[0135] The family tree service device (2120) may include, for example, a cloud server, and may be configured to include a DB (2120a) linked to the server. The server and the DB (2120a) may be connected to each other and communicate through an internal dedicated network such as an intranet. When using an internal dedicated network, unlike when using an external network such as the communication network (2110) of FIG. 1, separate data processing operations such as modulating / demodulating data, encoding / decoding may not be performed. The family tree service device (2120) can automatically generate and manage a family tree based on various types of data provided by users of the user terminal device (2100). The family tree service device (2120) can operate as a web program and mobile application platform that collects various information about family members from users and analyzes it to automatically generate and manage a family tree, and can perform genealogy digitization, AI-based photo restoration, external social data linkage, and automatic update operations to enhance user convenience.

[0136] For example, when a user inputs information, the family tree service device (2120) according to an embodiment of the present invention stores and manages the information in a central database such as the DB (2120a) of FIG. 1. Then, it can automatically connect relationships by executing an AI algorithm to analyze the data. In this process, for instance, a family tree can be created between grandparents, parents, oneself, and children. Furthermore, the family tree service device (2120) can synchronize additional data in real time through external integration with a third-party device (2130) of FIG. 1, and can create or strengthen a tree by integrating restored visual data and genealogical data. Moreover, it can operate to allow data to be viewed and modified on a user dashboard, i.e., a service screen.

[0137] More specifically, the family tree service device (2120) according to an embodiment of the present invention can perform operations such as automatic generation and visual provision of a family tree, data supplementation and sophisticated connection using AI and machine learning technologies, enhancement of personal information through photo and video restoration, integration of external data through social platform API linkage, digitization of genealogical data, and automatic analysis of blood relationships.

[0138] To automatically generate and visually provide a family tree, the family tree service device (2120) automatically analyzes the relationships of family members based on data entered by the user and provides them intuitively and visually. By moving away from the existing manual method and using an AI algorithm to automatically generate and visualize relationships, the user can easily explore the history of the family. Through this, the family tree can be expanded through collaboration not only among individuals but also among relatives, and intergenerational connectivity can be strengthened. In the case of the family tree, the visualization screen can display information on one side of the screen showing which era each family member lived in by year. When viewing the family tree, by displaying the chronological information, it is easy to confirm, for example, that the great-great-grandfather lived from the 1860s to 1930. Of course, this can be seen as based on the birth and death information of the great-great-grandfather in the family tree.

[0139] To supplement data and perform sophisticated connection operations using AI and machine learning technologies, the family tree service device (2120) supplements missing information that the user has not entered through AI-based analysis and refines duplicate data to increase reliability. For example, it analyzes surnames, ages, locations, etc., based on family member information entered by the user and automatically connects relationships with existing data. This allows the user to easily obtain a complete and systematic family tree. In the case of missing information, it is also possible to supplement the missing information by methods such as cross-verification from data provided regarding family members.

[0140] The family tree service device (2120) can enhance the visual information of family members by utilizing AI technology to restore old photo and video data and converting them to high resolution for the purpose of enhancing personal information through photo and video restoration. For example, black and white photos can be colorized or damaged parts restored, and integrated into the family tree to increase visual value. The restored images can also be operated to be used for avatar creation and metaverse integration. The restoration of photo or video data may be possible through the results of learning from training data that the AI ​​model has previously learned. For example, if the background in the photo is a Hanok from the 1970s, the AI ​​model has learned a vast amount of training data related to Hanoks from the 1970s, so it can analyze the shape of the Hanok, find the same Hanok, and restore the damaged parts of the Hanok in the photo accordingly. Of course, for damaged parts in photo or video frames, it is also possible to use the results of analyzing surrounding pixels.

[0141] Based on boundaries that utilize surrounding pixels, for example, if a roof tile part of a Hanok is damaged, the corresponding pixel can be restored by referencing other roof tiles. Of course, when restoring pixels, high-resolution photos or videos can be restored by restoring the pixels based on the pixel with the best resolution (e.g., color value or gradation value).

[0142] To enable the integration of external data through social platform API integration, the family tree service device (2120) can automatically collect and integrate data of the user and family members through API integration with social media and other external platforms. Through this, the user can reflect the latest information in the family tree without additional input. For example, social media profiles, photos, and location data can be retrieved and automatically added to the family tree. Recently, since it is possible to create family trees centered on small families, it is also possible to periodically update the recent activities of family members through integration with external platforms. For instance, if there is a promotion at work and information about it is posted on social media, that data can be reflected and added.

[0143] The family tree service device (2120) scans traditional genealogical documents to perform operations such as digitizing genealogical data and automatically analyzing blood relationships, and digitizes them using OCR technology and an AI-based translation tool. The digitized data can automatically analyze blood relationships and be integrated with existing tree data. Through this, traditional heritage and modern technology are fused, and the history of the family can be systematically managed in a digital environment. Here, translation may mean translating, for example, foreign languages ​​or Chinese characters into Hangul, but it may also mean extracting the content of a sentence through natural language processing using a Large Language Model (LLM). For example, since the official titles of ancestors are recorded in the genealogy, it is entirely possible to easily explain those titles. Explainable AI may be used in this process. Explainable AI may mean explaining the decision-making process of artificial intelligence in a form that humans can understand.

[0144] In addition, the family tree service device (2120) according to an embodiment of the present invention can perform various operations. For example, it may include operations such as registering digital assets (e.g., photos, NFTs, metaverse items, etc.) during one's lifetime, and when death is officially certified and a death certification signal (e.g., official documents, API, etc.) is received, automatically granting ownership and access rights to designated heirs according to a will / smart contract. It may also include operations such as digital asset information including forms such as metaverse avatars, NFTs, photos / videos, and blockchain tokens, and performing an internal platform arbitration algorithm at the time of a dispute when transferring inheritance rights. Meanwhile, the family tree service device (2120) can also function as a user dispute resolution device. It may perform operations such as using AI to analyze family relationship certificates, will information, and deceased account data submitted by heirs to determine whether there is a conflict, and recording and notarizing the dispute results based on blockchain. To reduce digital asset inheritance disputes, the pre-will registration, official death certification, blockchain history recording, and arbitration system are implemented as a single unit. This enables legal safety, minimizes disputes, and ensures long-term preservation, and can function as a mechanism that aligns with and materializes the purpose of digital heritage when using app / web / metaverse platforms.

[0145] First, the family tree service device (2120) according to an embodiment of the present invention operates as a web / app platform and can perform additional operations such as automatic node creation, connection, and verification in addition to the aforementioned data analysis and processing and family tree generation. First, conditional automatic connection and merging operations may be possible. An AI probability model (face recognition, text analysis, etc.) and a rule engine (family relationship logic) calculate a score (Confidence) to determine whether this node is identical to an existing person or a new person. If specific conditions (e.g., no conflict, high confidence) are satisfied, the node connection and merging are automatically confirmed through a proposal (accept / rejection) process to the user. It may also operate as a system that identifies duplicates through a conflict detection algorithm (birth year discrepancy, etc.) or by synthesizing multi-source data (photos, text, SNS, etc.). The family tree service device (2120) may also perform relearning operations based on user feedback (Manual Override). If a user corrects an incorrect connection, the AI ​​can learn from that case to reduce the error rate (Online Learning). The core operations can be seen as an AI model self-improvement structure (Adaptive or Online Learning) through feedback and automatic correction of duplicate / erroneous data. As mentioned above, the family tree service device (2120) can automatically organize people and family history in an optimal state by integrating AI restoration (e.g., sharpening blurry photos, restoring damaged documents), OCR (genealogy and character recognition), and de-duplication for refining and restoring duplicate photos / data. The key here is to implement image / document restoration, person recognition, and node integration as a single batch process. The family tree service device (2120) can also perform the operation of forming nodes for acquaintance relationships rather than blood relationships.Node connections resulting from user requests or faces presumed to be the same person in photos appear repeatedly in multiple photos (or at multiple points in time), and the AI ​​fails to match the blood relationship (e.g., parent-child, sibling) inference rules, or the fact that they are not family is confirmed through user feedback.

[0146] The family tree service device (2120) according to an embodiment of the present invention can perform a relationship strengthening operation based on repeated appearances. It can determine the frequency of appearance and the number of joint events in the relationship. For example, as the number of times the user's own node (or another person node) appears together in a photo increases, the relationship weight increases. It may be possible to distinguish between someone who appears once or twice by chance and a close friend who appears every time. Additionally, the family tree service device (2120) can determine an intimacy score. If the result of the platform's internal algorithm calculating a score based on the number of appearances, timing, and type of event exceeds a certain threshold, it is inferred that they are in a strong acquaintance relationship (friend). The family tree service device (2120) can confirm the relationship upon user approval. Through the process of notifying the user from automatic inference and the user accepting or rejecting, a system can be implemented where two nodes are connected as acquaintances even if they do not know each other's contact information. Of course, users can disable specific acquaintance nodes through notification rejection settings, and through permission settings, acquaintances may or may not be able to view the other person's family relationships.

[0147] Regarding metaverse platforms, memories are currently centered on photos and videos; however, in the future, services that provide new experiences for honoring the deceased by digitizing even the senses of smell and taste (e.g., digital perfumes or food recipes) may increase. Above all, the embodiments of the present invention can focus on the digital human heritage of the metaverse platform. Heritage can be expressed in various forms, such as people, objects, or memories (emotions), and the future world of Artificial General Intelligence (AGI) will inevitably take this form (a virtual world resembling reality). Furthermore, to implement this, technical, legal, and ethical systems (e.g., safety and security devices) must be established in advance.

[0148] In this regard, the family tree service device (2120) according to an embodiment of the present invention can perform (1) AI emotion and situation recognition and person avatar expansion operations. To this end, it can perform emotion recognition and emotion tagging operations. The AI ​​analyzes facial expressions and voice tones in photos and videos, adds emotion tags such as happiness, sadness, and emotion to nodes, and can collect and display only specific emotion themes (e.g., a collection of happy memories) in the metaverse space. The core of this can be seen as the analysis of emotion scores by person / photo, classification of emotion themes, and automatic curation.

[0149] The family tree service device (2120) can also perform operations related to (2) Imlaps avatars (person aging / past restoration). To do this, AI analyzes photo data from the past to the present to estimate changes in the person's appearance (e.g., childhood, middle age, etc.), and by moving a slider in the metaverse, the person avatar can be transformed according to age group. Additionally, it can perform family storytelling and timeline tour operations. For example, by combining genealogy / history and AI simulation, it can visualize and display virtual historical branching scenarios such as "What if this grandfather had made a different decision?" or "What if he hadn't evacuated during the war?". To do this, the family tree service device (2120) can implement technologies such as a branch-type story engine, family / history event data, and AI inference, which serve as educational and entertainment elements, allowing users to experience alternative history scenarios on a family-by-family basis. The key can be seen as implementing a time-series face synthesis / inference model into an avatar and having an automatic avatar generation logic for each age group.

[0150] In addition, the family tree service device (2120) according to an embodiment of the present invention can further perform operations related to (3) a chatbot / avatar of a deceased person (past person), (4) a group photo → 3D snapshot space, (5) a multi-user simultaneous access / memorial / commemorative event, (6) a five-sense expanded memory experience, (7) a memory correction through AI sentiment analysis, and (8) a permanent preservation system. (3) The operation related to a chatbot / avatar of a deceased person (past person) involves training an AI model with past data such as text, voice, and video to provide a function for a deceased person avatar to converse with a user in the metaverse. The key here is to extract the speech patterns and voice characteristics of a specific person and reproduce them as a conversational agent. (4) Through the operation related to a group photo → 3D snapshot space, individuals in a single group photo can be automatically segmented and recognized, and the background can be restored in 3D to reproduce the event at that point in time within the metaverse. The key here is to convert a 2D group photo into a 3D space, connect it to individual nodes (face recognition), and create virtual events. (5) Regarding multi-user simultaneous access and memorial / commemorative events, multiple people can log in at the same time and interact in real time (voice, chat, avatar actions) for events such as weddings, memorial days, and family parties. Through this, a metaverse event system can be implemented where multiple users share, edit, and play node-based memory content (photos, videos, BGM).

[0151] The family tree service device (2120) according to an embodiment of the present invention (6) can perform operations for a five-sense expanded memory experience. To this end, a memory restoration system based on scent and food recipes can be configured, and a virtual hug linked to tactile (glove) sensations can be made possible. Regarding the memory restoration system based on scent and food recipes, the deceased's favorite food recipes and specific scent information can be converted into data. In the metaverse space, recipe and food-related information can be visualized, and VR / AR taste and smell simulations can be performed. Regarding the virtual hug linked to tactile (glove) sensations, when meeting a family avatar in the metaverse and performing a virtual hug, a tactile feedback device is used to allow the sensation of a hug to be felt in reality. Of course, such feedback can also be controlled by a vibration motor installed in the user terminal device (2100). To this end, a tactile feedback suit / glove, real-time motion capture, and avatar synchronization technology within the metaverse can be implemented. Through this, far away

[0152] You can also recreate physical sensations to some extent with kind family and acquaintances, maximizing the sentiment of sharing memories.

[0153] The family tree service device (2120) can further perform (7) memory correction operations through AI emotional analysis. To this end, it can perform memory modification and correction simulation operations. The AI ​​recognizes that human memory is subjective and can be unstable, and re-edits the color tone, background music (BGM), and atmosphere of photos and videos according to the user's emotional state. For example, when the user is depressed, the memory video is edited to be brightly corrected, and when the user is happy, it is edited with calm background music to enable the reproduction of memories. Through this, a slightly philosophical and psychological healing concept can be applied, processing real-life memories for the purpose of emotional healing. Collaborative editing and remixing operations can be performed for memory correction operations. Multiple family members can simultaneously re-edit past photos and videos (e.g., filters, subtitles, music, etc.) and create a final version as a new memory to post on the metaverse. To this end, collaborative editing tools and real-time emotional and opinion reflection (e.g., voting, etc.) can be implemented.

[0154] Finally, the family tree service device (2120) can perform operations related to the (8) permanent preservation system. To ensure that memory data is not lost even if external hacking, errors, or data center fires occur, an IPFS (Distributed File System) or encryption distributed storage and self-recovery protocol can be implemented on multiple nodes. Through this, the concept of a digital vault, which aims for long-term preservation for 2100 or 22200 years, can be implemented in the metaverse. In addition, for automatic notifications and continuous donations to descendants, even if the generation changes and the account is neglected, the AI ​​can send family legacy backup guidance to descendants at regular intervals. For example, when the great-grandchild generation appears, automatic email / metaverse message sending is provided. Through this, a truly permanent management structure can be implemented.

[0155] We would like to further examine how the family tree service device (2120) can operate as an automatic digital estate inheritance management and dispute mediation system. It can manage a user's digital content (e.g., photos, videos, social media records, NFTs, etc.) as a digital estate by centralizing it on a single platform (web / app / metaverse), allow the user to set a digital will during their lifetime, or allow heirs to inherit the assets without dispute after confirming official procedures (e.g., death certificate, etc.) after death, and, in the event of a dispute, allow it to be processed quickly and transparently through a centralized / decentralized dispute resolution module.

[0156] For the above operation, the family tree service device (2120) can perform operations such as user pre-registration and death certification trigger in relation to digital will and death certification linkage operations. The former means setting whether to use the Digital Will function when signing up for a web / app / metaverse platform, and specifically specifying, for example, to whom and under what conditions one's metaverse avatar, photos, videos, and NFT assets will be inherited upon death, and storing this information (will) in an encrypted form on a blockchain or a separate secure DB. The latter, namely the death certification trigger operation, means that the platform can automatically assign a death status flag to the corresponding user account when a death certificate or death report is received by linking APIs with the government (administrative agency), hospital, insurance company, etc. The automatic processor can operate in a sequence such as activating the digital estate inheritance module when death is confirmed.

[0157] Additionally, the family tree service device (2120) performs inheritance distribution and automatic access rights switching operations for the above operations, and to this end, it may perform operations such as conditional smart contracts, multi-signatures, or multiple approvals and rights switching. Regarding conditional smart contracts, a smart contract is created in advance according to the will, and when death is confirmed, the contract transfers ownership and access rights of property (e.g., digital assets) to specific heirs. For example, detailed settings may be possible, such as disclosing some of my social media records only to my spouse, or sharing sensitive information in photos only with my children. Regarding multi-signatures or multiple approvals, when there are multiple family members (or legal heirs), it can be processed in a structure of multiple approvals (e.g., one wife, two children, one legal representative, etc.), and configured so that final access is allowed only with consent. Regarding the transfer of authority, ownership and management rights of a deceased person's avatar in the metaverse platform are automatically transferred to the heir, and the heir can choose to maintain, delete, or make public the corresponding avatar and memory space.

[0158] The family tree service device (2120) can execute a dispute mediation module to perform operations such as automatic arbitration (or stages) when a dispute occurs, AI-based analysis of relationships and supporting documents, and procedures for agreement and objection. In the automatic arbitration stage when a dispute occurs, if a dispute arises between heirs A and B regarding "who should actually have more rights," the platform can initiate an arbitration process. Here, on-chain arbitration may mean that a pre-agreed group of arbitrators (Arbitration Committee) or the platform's own algorithm comprehensively reviews (1) official documents, (2) will records left by the user, etc., and draws a conclusion. Regarding the AI-based analysis of relationships and supporting documents, for example, the AI ​​can automatically identify genealogical records, family registers, marriage and divorce documents, etc., to determine whether A is the spouse and whether B is not the legal representative, and record the final conclusion on the blockchain to transparently record who inherited when and through what procedures. Regarding settlement and objection procedures, the system can be configured to allow for escalation to actual court proceedings (notarization) if the outcome of the dispute is unsatisfactory. However, utilizing the arbitration procedures provided by the platform enables faster resolution and reduces the likelihood of resorting to external litigation.

[0159] In addition, the family tree service device (2120) can perform operations related to node (e.g., digital assets) encryption and access levels to enhance privacy and security, as well as operations related to compliance with international regulations such as the General Data Protection Regulation (GDPR) and the Personal Information Protection Act (CCPA). Regarding the former, the deceased's data (photos, messages, personal records) may be encrypted and stored in advance, and even heirs may be required to undergo separate authentication to view sensitive information. Compliance with personal information protection and international regulations such as the GDPR and CCPA may be required. Data is disclosed to heirs only within the scope agreed upon by the deceased during their lifetime, and the scope of disclosure / non-disclosure may be provided according to the settings of the will.

[0160] The third-party device (2130) may include various types of devices that API-link with the family tree service device (2120) of FIG. 5.

[0161] The third-party device (2130) may include, for example, an SNS server that provides SNS services. The family tree service device (2120) can automatically retrieve data related to family members through a social media API and use SNS profile pictures, posts, and location information to correct the accuracy and richness of the data. In other words, the data on family members provided by the user terminal device (2100) at the beginning of the service may be insufficient below a standard (e.g., missing photos of specific family members), but when the data exceeds the standard (e.g., supplementing photos of family members whose photos are missing), the completeness of the family tree and the data provided in relation thereto can be seen to increase. The third-party device (2130) can be involved in this operation.

[0162] Of course, the third-party device (2130) may include a server or computer operated by a company that produces genealogies. If genealogy data is needed, it is also possible to obtain it from the company's device.

[0163] In addition to the above, details regarding the user terminal device (2100), communication network (2110), family tree service device (2120), and third-party device (2130) of FIG. 1 will continue to be covered later, so the details will be replaced by those details.

[0164] FIG. 6 is a block diagram illustrating the detailed structure of the family tree service device of FIG. 1, and FIG. 7 is a block diagram illustrating the detailed structure of the family tree service unit of FIG. 6.

[0165] As illustrated in FIG. 6, the family tree service device (2120) of FIG. 5 according to an embodiment of the present invention includes some or all of a communication interface unit (22200), a control unit (2210), a family tree service unit (2220), and a storage unit (2230).

[0166] Here, "including some or all" means that some components, such as the storage unit (2230), may be omitted to configure the family tree service device (2120) of FIG. 1, or that some components, such as the family tree service unit (2220), may be integrated into other components, such as the control unit (2210). To facilitate a sufficient understanding of the invention, it is described as including all components.

[0167] The communication interface unit (22200) communicates with the user terminal device (2100) and the third-party device (2130), respectively, via the communication network (2110) of FIG. 1. During the process of performing communication, the communication interface unit (22200) performs operations such as modulation / demodulation, encoding / decoding, and muxing / demuxing, and can perform operations such as scaling to convert the resolution. Since this is obvious to those skilled in the art, further explanation is omitted.

[0168] The communication interface unit (22200) can receive family member information and family relationship data for creating a family tree in accordance with a service request from the user terminal device (2100). Here, the family member information may include basic user information such as name, gender, and age, and detailed information such as occupation, personality, voice, and memories may be received. Additionally, photos or videos may be provided as family relationship data. Here, the photos or videos may include photos or videos of a family tree.

[0169] Additionally, the communication interface unit (22200) can provide a visualized screen of the family tree under the control of the control unit (2210), and of course, the visualized screen can be a graphic screen. The visualized screen can be provided in the form of an scalable interactive view, a 3D tree view, a timeline view, etc. For example, the control unit (2210) can receive a control signal from an interface with a user on the screen and process and provide the screen based on the signal. In addition, the communication interface unit (200) can also provide data of a visualized screen that reflects the collected data of family members collected through communication with the third-party device (2130) of FIG. 1 into the family tree. The communication interface unit (22200) can receive data collected through API linkage with the third-party device (130).

[0170] The control unit (2210) may include a processor such as a CPU, MPU, or GPU, and may further include memory such as RAM. It is also possible for the processor and memory to be configured as a single chip, such as an IC chip. The control unit (2210) performs overall control operations of the communication interface unit (22200), family tree service unit (2220), and storage unit (2230) of FIG. 2. For example, when there is a service request for family tree creation from the user terminal device (2100) of FIG. 1, the control unit (2210) may receive relevant information from the communication interface unit (22200) and execute a graphic (GUI) program to provide a visualization screen mounted on the family tree service unit (2220). Of course, in this process, data stored in the DB (2120a) of FIG. 1 may be retrieved and inserted into the corresponding visualization screen to be provided.

[0171] The control unit (2210) can perform various other operations. For example, the control unit (2210) can generate a family tree to be provided to the user terminal device (2100) of FIG. 1, or receive family relationship data such as black-and-white photos or videos to be reflected in the generated family tree. This data can be temporarily stored in the storage unit (2230), retrieved, and then requested to the family tree service unit (2220) for data analysis. Then, through data analysis, if the family tree service unit (2220) applies AI restoration technology to restore old photos and videos to color and outputs high-resolution data, this can be systematically classified, stored, and managed in the DB (2120a) of FIG. 1 for each user.

[0172] The family tree service unit (2220) performs operations to create and manage a family tree based on user data.

[0173] The family tree service department (2220) can primarily perform operations such as providing automatic generation and visualization of family trees, supplementing data and sophisticated linking using AI and machine learning technologies, enhancing personal information through photo and video restoration, integrating external data through social platform API linkage, digitizing genealogical data, and automatically analyzing blood relationships. Of course, it can also perform additional operations such as family memory gallery and family record enhancement. The family memory gallery operation may mean providing a tab that allows users to easily access restored photos and videos organized in chronological order, and the family record enhancement operation means an operation in which data is enhanced by the user requesting relatives to add missing data of a specific member, either automatically designated or manually designated.

[0174] For example, on the visualized screen of the family tree, buttons such as a request for relative data may be activated and displayed around a specific member. Accordingly, the member or the representative of the family members can select the button to request missing data of the specific member from a designated relative. Through this, the relative's user terminal device (2100) may receive a relevant push message via the app and provide the missing data. For example, if the family tree is created and managed up to the father and mother, a typical case would be to request missing data from the father's siblings (e.g., uncles, aunts, etc.) or the mother's siblings (e.g., maternal aunts, etc.) based on genealogical data. Additionally, one can request missing data from their own siblings in their own case. The family tree service unit (2220) analyzes the data of family members and, when missing data is determined, activates it in the form of a button on the screen; when the missing data request button is selected, the data can be requested in the form of a push message using the app. Of course, regarding the button on that screen, it is entirely possible to display user information about which relatives possess missing data. In this case, it is also entirely possible to request the data through a separate messenger-based chat service.

[0175] The family tree service unit (2220) may be configured in various ways, but as illustrated in FIG. 3, it may include some or all of the information input and management unit (2300), family tree creation and visualization unit (2310), social media and external data linkage unit (2320), photo and video restoration unit (2330), genealogy digitization and translation unit (2340), and data security and access control unit (2350). Furthermore, it may further include a manager that performs control functions in the form of software that controls the above components in response to a request from the control unit (2210) of FIG. 6. Here, "including some or all" is not significantly different from the meaning above. The components of FIG. 7 constituting the family tree service unit (2220) may be composed of hardware (H / W) modules, software (S / W) modules, or a combination thereof.

[0176] The information input and management unit (2300) includes an information input and management module and can perform operations such as standardized data input and automatic data supplementation through the execution thereof. For data input operations, a form can be provided that allows the user to input basic information (e.g., name, gender, age, etc.) and detailed information (e.g., occupation, personality, voice, memories, etc.). For automatic data supplementation, missing information can be automatically supplemented based on kinship data by utilizing an AI-based data cross-verification algorithm. For example, between relatives, it is possible to determine whether the people in the photos are the same person through facial feature points, and through this, the kinship relationship can be recognized and the data of the members can be cross-verified with each other. The information input and management unit (2300) can receive personal information (e.g., name, date of birth, gender, etc.) provided by the user and data of family members via the web or app. Additionally, it can provide an interface for inputting sophisticated data such as photos, voice files, and memory records. Through data analysis and processing operations, input information can be analyzed using an AI-based algorithm to automatically supplement missing data, remove duplicate input data, and review consistency with existing data.

[0177] The family tree creation and visualization unit (2310) executes the family tree creation and visualization module, thereby performing operations such as automatic relationship connection and providing multiple views. For the automatic relationship connection operation, it can analyze input data to automatically connect relationships between family members and create a tree. In addition, for providing multiple views, it can operate to allow the user to select various visualizations, such as an scalable interactive view, a 3D tree view, and a timeline view. The family tree creation and visualization unit (2310) creates a tree by automatically connecting family members based on the results of the relationship analysis, and the tree supports multiple views (e.g., 2D plane, 3D tree, timeline, etc.) so that the user can navigate in a way preferred by the user.

[0178] The social media and external data linkage unit (2320) executes a social media and external data linkage module and can perform API-based linkage to automatically collect family-related data through linkage with SNS, email, and other social platforms, as well as data refinement and integration operations to structure the collected data, remove duplicates and errors, and integrate it into a database. That is, data integration can be performed after the collected data is preprocessed. The social media and external data linkage unit (2320) automatically retrieves data related to family members through a social media API for SNS linkage and data enhancement, and ensures the accuracy and richness of the data by utilizing SNS profile pictures, posts, and location information.

[0179] The photo and video restoration unit (2330) executes a photo and video restoration module and implements AI restoration technology to restore old photos and videos to color and perform upscaling operations to high resolution. Of course, the restoration of old photos or videos is based on the results of training data that the AI ​​model has previously learned, but it is also possible to restore specific parts using surrounding pixels in a single video frame. In addition, to enhance visual data, the restored images can be automatically added to profile data to strengthen the tree. The photo and video restoration unit (2330) restores old black-and-white photos to color using AI technology through AI photo restoration, increases the resolution of the photos, and repairs damaged parts to enhance the quality of visual materials, and the restored photos can be stored in a family tree database and used for avatar creation and memory implementation.

[0180] The genealogy digitization and translation unit (2340) can scan genealogy documents, extract text using OCR technology, and construct a digital genealogy through AI-based translation and data structuring to supplement the family tree. To this end, the genealogy digitization and translation unit (2340) can execute a genealogy digitization and translation module to perform OCR-based scanning, automatic translation, and structuring operations. Through OCR-based scanning, the genealogy can be scanned and text can be extracted using AI-based OCR. When AI is applied, errors in the text extracted by OCR can be reduced. Additionally, the extracted data can be translated for automatic translation and structuring to convert it into a digital genealogy and integrated into the tree. Here, translation may include semantic analysis, and a large language model (LLM) for natural language processing may be used for this purpose.

[0181] The data security and access control unit (2350) executes a data security and access control module, and all data can be stored and transmitted using the AES-256 encryption algorithm for data encryption. Additionally, for user authority management, it can provide an interface that allows users to set access rights and restrictively disclose specific data. Various methods may be used for user authority management, such as fingerprint recognition, pattern setting, or password setting.

[0182] Of course, in addition to this, the family tree service unit (2220) can perform data synchronization operations or metaverse linkage operations.

[0183] For example, data stored in a central database, such as the DB (2120a) of FIG. 5, can be linked with the metaverse platform. Through data synchronization, real-time synchronization is achieved so that input information or changes can be reflected immediately. In the case of metaverse linkage, data of family members is retrieved from the database to create 3D avatars and virtual spaces. An environment can be provided within the metaverse platform where the user and family members can interact. Since the family tree service unit (2220) may directly provide metaverse services, the embodiments of the present invention will not be specifically limited to any one form. For example, since parts related to automatic digital estate inheritance management or dispute mediation have been sufficiently explained above, detailed information will be replaced by those details.

[0184] In summary, the family tree service unit (2220) according to an embodiment of the present invention enables real-time data transmission between the data input module and the AI ​​processing module through interaction between modules, allows the external API integration module to refine data and integrate it into the main database, and allows the photo restoration module to store and visualize results based on user input data. To enhance security, all data can be securely transmitted via AES-256 encryption and SSL, and users can set access rights for specific information, thereby enabling the protection of sensitive data. Furthermore, in terms of scalability, an API-based architecture can be used to easily integrate with other social platforms, cloud storage, or additional services.

[0185] The storage unit (2230) of Fig. 6 can temporarily store various types of information or data processed under the control of the control unit (2210).

[0186] Here, since information and data are used interchangeably in practice, the concept of the terms will not be specifically limited. In an embodiment of the present invention, for example, the family member information provided from the user terminal device (2100) of FIG. 1 may include basic information such as date of birth as user information, and may also include data such as photos or videos as family relationship data. The storage unit (230) may temporarily store the family member information or family relationship data and then provide it to the family tree service unit (2220) so that data analysis can be performed.

[0187] In addition to the above, the communication interface unit (22200), control unit (2210), family tree service unit (2220), and storage unit (2230) of FIG. 6 can perform various operations, and other details have been sufficiently explained above, so they will be replaced by those details.

[0188] According to an embodiment of the present invention, the communication interface unit (22200), control unit (2210), family tree service unit (2220), and storage unit (2230) are composed of hardware modules that are physically separated from each other, but each module may store software for performing the above operations internally and execute it. However, since the software is a set of software modules and each module can be formed as hardware, the configuration is not specifically limited to software or hardware. For example, the storage unit (2230) may be storage or memory which is hardware. However, since it is also possible to store information in a software repository, the above is not specifically limited.

[0189] Meanwhile, as another embodiment of the present invention, the control unit (2210) may include a CPU and memory and may be formed as a single chip. The CPU includes a control circuit, an arithmetic logic unit (ALU), an instruction interpretation unit, and a registry, and the memory may include RAM.

[0190] The control circuit can perform control operations, the operation unit can perform operations on binary bit information, and the instruction interpretation unit can perform operations to convert high-level language into machine language and machine language into high-level language, including an interpreter or compiler, and the registry can be involved in software data storage. According to the above configuration, for example, at the beginning of the operation of the family tree service device (120), the program stored in the family tree service unit (2220) can be copied and loaded into memory, i.e., RAM, and then executed, thereby rapidly increasing the data operation processing speed. In the case of a deep learning model, it may be loaded into GPU memory instead of RAM and executed by using the GPU to accelerate the execution speed.

[0191] Figure 8 is a flowchart showing the operation process of the family tree service device of Figure 5.

[0192] For convenience of explanation, referring to FIG. 8 together with FIG. 5, the family tree service device (2120) of FIG. 5 according to an embodiment of the present invention communicates with a user terminal device (2100) that uses a family tree service to form and manage relationships with family members (S2400). In the case of a PC-based terminal device, web services can be provided through communication, but in the case of a mobile-based terminal device, app services can be provided through communication. In the case of web services, email services, text message (SMS) services, and messenger-based chat services may be used together for notification during the process of providing services. On the other hand, in the case of app services, notifications may be provided in the form of push messages.

[0193] In addition, the family tree service device (2120) automatically generates a family tree based on family member information and family relationship data such as photos or videos provided by the user terminal device (2100) for generating the family tree, and when there is a lack of family member information or family relationship data during the automatic generation of the family tree, it can automatically generate the family tree by applying artificial intelligence (AI) to supplement the lacking information or data (S2410).

[0194] For example, missing information or data can be supplemented in various ways. In the case of insufficient information, it can be supplemented by cross-verifying data among family members (or relatives). Furthermore, regarding photo or video data, restoration can be performed based on training data (e.g., Hanok from the 1970s) learned by the AI, and if a specific part of a photo is missing, it is entirely possible to restore the photo using surrounding pixels. However, when converting black-and-white photos to color, data supplementation may be possible by converting them to color based on training data related to specific trees, for instance, in the case of trees.

[0195] In addition, the family tree service device (2120) can perform various other operations. For example, a family memory gallery or a family record reinforcement operation may be representative. In the case of a family memory gallery, restored photos and videos can be organized chronologically to provide tabs that users can easily access. Chronological order can be based on birth and death information of the members constituting the family tree. For example, sorting the school years of family members in the order of elementary, middle, high school, and college may be representative. It is also possible to organize them in the order of years. In cases where time information is missing in photos or videos, time information such as years can be determined through training data using various objects in the photos or videos, such as clothing styles or building styles. In the case of a family record reinforcement operation, a reinforcement button can be activated and displayed on the visualized screen, that is, in the part adjacent to the member constituting the family tree, to reinforce missing data. In this case, a request to reinforce missing data can be transmitted based on the contact information or URL information of a relative that is automatically matched to the reinforcement button. Of course, it is entirely possible to supplement data by manually entering contact information. A typical example is entering a phone number for sending a text message (SMS). For instance, even in the case of relatives, if they are subscribed to the service according to the embodiment of the present invention, they may be notified via a push message through an app that a specific relative has requested supplementary data. Of course, since the service can be implemented in various ways, the embodiment of the present invention will not be specifically limited to any single form.

[0196] In addition to the above, the family tree service device (2120) of FIG. 5 can perform various operations, and other details have been sufficiently explained above, so they will be replaced with those details.

[0197] Although all components constituting an embodiment of the present invention have been described as being combined or operating as a single unit, the present invention is not necessarily limited to such an embodiment. That is, within the scope of the purpose of the present invention, all components may be selectively combined and operated in one or more ways. Furthermore, while all components may each be implemented as a single independent piece of hardware, they may also be implemented as a computer program having a program module that performs some or all of the combined functions on one or more pieces of hardware by selectively combining some or all of the components. The codes and code segments constituting the computer program can be easily inferred by those skilled in the art of the present invention. An embodiment of the present invention may be implemented by storing such a computer program on a non-transitory computer-readable medium, reading it, and executing it by a computer.

[0198] FIG. 9 is a diagram showing a metaverse service system according to an embodiment of the present invention.

[0199] As illustrated in FIG. 9, a metaverse service system (3090) according to an embodiment of the present invention includes some or all of a user terminal device (3100), a communication network (3110), a metaverse service device (3120), and a third-party device (3130).

[0200] Here, "including some or all" means that some components, such as a third-party device (3130), may be omitted to configure the metaverse service system (3090), or that some or all of the components constituting the metaverse service device (3120) may be integrated into a network device (e.g., gateway, wireless exchange device, etc.) constituting the communication network (3110), and is described as including all to facilitate a sufficient understanding of the invention.

[0201] A user terminal device (3100) can access the metaverse service device (3120) of FIG. 9 to use the family data-based metaverse service according to an embodiment of the present invention. The user terminal device (3100) may include not only PC-based terminal devices such as desktop computers and laptop computers for using the service, but also mobile-based terminal devices such as smartphones, tablet PCs, and wearable devices worn by the user on the wrist, etc. Through a PC-based terminal device, one can access the metaverse service device (3120) of FIG. 9 to use web services, and through a mobile-based terminal device, one can access the metaverse service device (3120) to use application (hereinafter, App.) services. If notifications related to the service are provided in the form of push messages through the App service, in the case of a PC-based terminal device using web services, various additional services such as notifications may be provided through email, text message service (3MS), messenger-based chat service, etc.

[0202] The family data-based metaverse service used by the user terminal device (3100) is a service that allows the user and family members to interact emotionally in a metaverse service device (3120) that operates as a metaverse platform based on family data collected from the user of the user terminal device (3100). In this process, it may refer to a service that expands the use of family-centered data through a security and approval system (or service) to enable the user to emotionally connect with the person they wish to see. In other words, the user of the user terminal device (3100) can experience an immersive emotional experience encompassing the past, present, and future through the metaverse service by utilizing family relationship data and memories. For example, when missing deceased parents, the user can access the metaverse service to return to their childhood days spent with their parents, and, for instance, select the corresponding year on the service screen to experience that time in a virtual space through avatars and places of memories. Of course, since this experience is based on family data provided by the user, it may differ somewhat from reality, but this can be changed as much as the amount of family data provided by the user. In other words, as the amount of family data provided increases, it can match the experiences of the era in which the user actually lived. The metaverse service device (3120) creates a realistic avatar based on family member data, such as appearance, voice, personality, and memories, and recreates places of memories in 3D, thereby strengthening the emotional connection of the user, and the user terminal device (3100) can use this type of service.

[0203] The user terminal device (3100) can connect to the metaverse service device (3120) of FIG. 9 and provide family data when initially using the service. For example, it is possible to provide data by year from the time of one's birth to the present on the service screen provided by the metaverse service device (3120). For instance, in the case of direct ancestors, family data regarding one's father, mother, and even grandfather or grandmother can be provided. Accordingly, even in an era before one was born, one can experience the time when one's father and mother lived with one's grandfather and grandmother in the virtual space of the metaverse. Of course, in the case of direct ancestors, a family tree can be generated based on family data such as a genealogy provided by the user, and it is possible to provide data related to family members within that family tree. Of course, the metaverse service device (3120) of FIG. 9 can provide a service screen of a designated format, i.e., a layout, to collect family data from the user terminal device (3100), and through this, family data can be collected. Representative examples of layout screens include screens that distinguish by year.

[0204] The communication network (3110) can be configured in various forms. The communication network (3110) may include both wired and wireless communication networks. For example, a wired or wireless internet network may be used or interconnected as the communication network (3110). Here, the wired network includes internet networks such as cable networks or public telephone networks (PSTN), and the wireless communication network includes CDMA, WCDMA, GSM, EPC (Evolved Packet Core), LTE (Long Term Evolution), WiBro networks, and public or private wireless networks based on LPWA (Low Power Wide Area) used for the application of Internet of Things (IoT) technology recently. Of course, the communication network (3110) according to the embodiment of the present invention is not limited thereto and may be used as an access network for a next-generation mobile communication system to be implemented in the future, for example, a cloud computing network under a cloud computing environment, a 5G network, a 6G network, etc. For example, if the communication network (3110) is a wired communication network, the access point within the communication network can be connected to the exchange office of the telephone company, etc., but if it is a wireless communication network, it can be connected to the SGSN or GGSN (Gateway GPRS Support Node) operated by the telecommunications company to process data, or connected to various relay stations such as BTS (Base Transceiver Station), NodeB, e-NodeB to process data.

[0205] The communication network (3110) may include access points. Access points may include small base stations, such as femto or pico base stations, which are often installed within a building. Here, femto or pico base stations may be classified according to the maximum number of user terminal devices (3100), etc., that can be connected. Of course, user terminal devices (3100), etc., may include a short-range communication module for performing short-range communication such as Zigbee and Wi-Fi. Access points may use TCP / IP or RTSP (Real-Time Streaming Protocol) for wireless communication.

[0206] Here, short-range communication can be performed using various standards other than Wi-Fi, such as Bluetooth, Zigbee, infrared (IrDA), RF (Radio Frequency) such as UHF (Ultra High Frequency) and VHF (Very High Frequency), and ultra-wideband communication (UWB). Accordingly, the access point can extract the location of a data packet, designate the best communication path for the extracted location, and transmit the data packet along the designated communication path to the next device, such as a metaverse service device (3120). The access point can share multiple lines in a typical network environment and may include, for example, routers, repeaters, and relays.

[0207] The metaverse service device (3120) may be configured to include, for example, a cloud server and a DB (3120a) linked to the server. The server and the DB (3120a) communicate by being connected via an internal dedicated network such as an intranet, and since it is not an external network like the communication network (3110) of FIG. 9, data processing operations such as modulation / demodulation and encoding / decoding may not be performed. However, in the case of sensitive personal information, operations such as encryption / decryption may be performed to prevent risks such as hacking. The DB (3120a) can systematically classify, store, and manage user-specific family data processed by the server. For example, when there is a request for the metaverse service from the user terminal device (3100) of FIG. 9, the user's family data can be retrieved from the DB (3120a) and inserted into a designated service screen to provide the service.

[0208] The metaverse service device (3120) according to an embodiment of the present invention can provide a metaverse service that offers an immersive emotional experience encompassing the past, present, and future by utilizing family relationship data and memories. For example, in the case of the future, a virtual space can be created by prediction using artificial intelligence technology, and a metaverse service that can give hope to the user can be provided accordingly. Of course, if such a future becomes the present again, it is entirely possible to provide a service by modifying the virtual space and avatars related to the previously created future based on family data related to the present. The metaverse service device (3120) can be seen as strengthening emotional connections between users by creating realistic avatars based on family member data, such as appearance, voice, personality, and memories, and by reproducing places of memories in 3D. For example, the user of the user terminal device (3100) can provide various family data, such as family photos and videos from the 1970s.

[0209] Therefore, the metaverse service device (3120) can recreate the background of a photograph from the 1970s in 3D and create family members in the photograph in the form of avatars to provide metaverse services. Of course, it is desirable to reflect the characteristics of men, women, and children when implementing the avatars. In addition, when creating avatars, data such as photos or videos can be used to create avatars by applying artificial intelligence technology. A generative AI model may be used as the artificial intelligence model. Also, for natural language processing, it is entirely possible to use a large language model (3LLM), etc.

[0210] As will be discussed in detail later, the metaverse service device (3120) can perform operations related more specifically to the creation of family data-based avatars, 3D reproduction of places of memories, expansion of worldviews based on security and consent, provision of emotion-centered immersive experiences, management of digital family heritage, and a global social metaverse connection hub. Through this, it can share family memories or places with other users through a global memory exchange program, promote cultural understanding, support seamless interaction with users of other languages ​​through translation and interpretation AI, and enable family avatars of various nationalities to interact in the metaverse on specific anniversaries (3 e.g., World Family Day) through global events and communities. By expanding the family data-based platform into a global community, the value of inspiration and connection can be expanded globally.

[0211] The metaverse service device (3120) supports emotional interaction by creating a 3D avatar, or virtual person, that reflects the appearance, voice, personality, and behavioral patterns of family members for the creation of family data-based avatars. Additionally, through AI-based learning, it can implement natural behavior and conversation patterns that reflect the characteristics of family members. The metaverse service device (3120) recreates family memory places in a 3D virtual space based on photos, videos, GPS data, etc. for the 3D recreation of memory places. It may also support the use of AI photogrammetry technology to supplement incomplete data and to set user-customized environments (e.g., weather, time of day, etc.). By applying artificial intelligence technology, it is possible to supplement incomplete data based on training data, and in the case of photos, it is also possible to restore pixels in lost areas by utilizing surrounding pixels.

[0212] Additionally, the metaverse service device (3120) can configure a security and consent system that enables interaction with people the user wishes to see, beyond family-centered data, for the expansion of a security and consent-based worldview. It can control access to personal data and guarantee the user's privacy through invitation-based connections. Furthermore, the metaverse service device (3120) can analyze the user's real-time emotions through emotion sensing technology to provide an emotion-centered immersive experience, and based on this, provide customized moods for the virtual environment or reactions of the avatar. Of course, emotions can be utilized by analyzing emotions in photos or videos. For instance, the face of a human object can be extracted from an image, and facial expressions such as smiling or crying can be determined using a template-based method or an object expression classifier. It can enhance immersion by recreating memories with family members or creating new emotional stories. For the management of digital family heritage, the metaverse service device (3120) can provide a platform that allows family data to be stored and managed as digital heritage and passed down to future generations. This can create new value by connecting family history across generations. The metaverse service device (3120) can expand the family-centered metaverse into a global social platform to operate as a global social metaverse connection hub, thereby supporting interaction between users of various cultures and backgrounds.

[0213] In addition, the metaverse service device (3120) according to an embodiment of the present invention can perform various operations. For example, it may include operations such as registering digital assets (e.g., photos, NFTs, metaverse items, etc.) during one's lifetime, and when death is officially certified and a death certification signal (e.g., official documents, API, etc.) is received, automatically granting ownership and access rights to designated heirs according to a will / smart contract. It may also include operations such as digital asset information including forms such as metaverse avatars, NFTs, photos / videos, and blockchain tokens, and performing an internal platform arbitration algorithm at the time of a dispute when transferring inheritance rights. Meanwhile, the metaverse service device (3120) can also function as a user dispute resolution device. It may perform operations such as using AI to analyze family relationship certificates, will information, and deceased account data submitted by heirs to determine whether there is a conflict, and recording and notarizing the dispute results based on blockchain. To reduce digital asset inheritance disputes, the pre-will registration, official death certification, blockchain history recording, and arbitration system are implemented as a single unit. This enables legal safety, minimizes disputes, and ensures long-term preservation, and can function as a mechanism that aligns with and materializes the purpose of digital heritage when using app / web / metaverse platforms.

[0214] First, the metaverse service device (3120) according to an embodiment of the present invention operates as a web / app platform and can perform additional operations such as automatic node creation, connection, and verification in addition to the aforementioned data analysis and processing and family tree generation. First, conditional automatic connection and merging operations may be possible. An AI probability model (face recognition, text analysis, etc.) and a rule engine (family relationship logic) calculate a score (Confidence) to determine whether this node is identical to an existing person or a new person. If specific conditions (e.g., no conflict, high confidence) are satisfied, the node connection and merging are automatically confirmed through a proposal (accept / rejection) process to the user. It may also operate as a system that identifies duplicates through a conflict detection algorithm (birth year discrepancy, etc.) or by synthesizing multi-source data (photos, text, SNS, etc.). The metaverse service device (3120) may also perform relearning operations based on user feedback (Manual Override). If a user corrects an incorrect connection, the AI ​​can learn from that case to reduce the error rate (Online Learning). The core operations can be seen as an AI model self-improvement structure (Adaptive or Online Learning) through feedback and automatic correction of duplicate / erroneous data. As mentioned above, the metaverse service device (3120) can automatically organize people and family histories in an optimal state by integrating AI restoration (e.g., sharpening blurry photos, restoring damaged documents), OCR (genealogy and character recognition), and de-duplication for refining and restoring duplicate photos / data. The key here is to implement image / document restoration, person recognition, and node integration as a single batch process. The metaverse service device (3120) can also perform the operation of forming nodes for acquaintance relationships that are not blood relationships.Node connections resulting from user requests or faces presumed to be the same person in photos appear repeatedly in multiple photos (3 or multiple viewpoints), and the AI ​​fails to match the blood relationship (e.g., parent-child, sibling) inference rules, or the fact that they are not family is confirmed through user feedback.

[0215] The metaverse service device (3120) according to an embodiment of the present invention can perform a relationship strengthening operation based on repeated appearances. It can determine the frequency of appearance and the number of joint events in this relationship. For example, as the number of times a user's own node (or another person node) appears together in a photo increases, the relationship weight increases. It may be possible to distinguish between someone who is photographed once or twice by chance and a close friend who appears every time. Additionally, the metaverse service device (3120) can determine an intimacy score. If the result of an internal platform algorithm calculating a score based on the number of appearances, timing, and type of event exceeds a certain threshold, it is inferred that they are in a strong acquaintance relationship (friend). The metaverse service device (3120) can confirm the relationship upon user approval. Through the process of automatic inference, notification to the user, and user acceptance / rejection, a system can be implemented where two nodes are connected as acquaintances even if they do not know each other's contact information. Of course, users can disable specific acquaintance nodes through notification rejection settings, and through permission settings, acquaintances may or may not be able to view the other person's family relationships.

[0216] Regarding metaverse platforms, memories are currently centered on photos and videos; however, in the future, services that provide new experiences for honoring the deceased by digitizing even the senses of smell and taste (e.g., digital perfumes or food recipes) may increase. Above all, the embodiments of the present invention can focus on the digital human heritage of the metaverse platform. Heritage can be expressed in various forms, such as people, objects, or memories (emotions), and the future world of Artificial General Intelligence (AGI) will inevitably take this form (a virtual world resembling reality). Furthermore, to implement this, technical, legal, and ethical systems (e.g., safety and security devices) must be established in advance.

[0217] In this regard, the metaverse service device (3120) according to an embodiment of the present invention can perform (1) AI emotion and situation recognition and person avatar expansion operations. To this end, it can perform emotion recognition and emotion tagging operations. The AI ​​analyzes facial expressions and voice tones in photos and videos, adds emotion tags such as happiness, sadness, and emotion to nodes, and can collect and display only specific emotion themes (e.g., a collection of happy memories) in the metaverse space. The core of this can be seen as the analysis of emotion scores by person / photo, classification of emotion themes, and automatic curation.

[0218] The metaverse service device (3120) (2) may also perform operations related to im-lapse avatars (person aging / past restoration). To do this, AI analyzes photo data from the past to the present to estimate changes in the person's appearance (e.g., childhood, middle age, etc.), and by moving a slider in the metaverse, the person avatar can be transformed according to age group. Additionally, it may perform family storytelling and timeline tour operations. For instance, by combining genealogy / history and AI simulation, it may visualize and display virtual historical branching scenarios such as "What if this grandfather had made a different decision?" or "What if he hadn't evacuated during the war?". To do this, the metaverse service device (3120) can implement technologies such as a branch-type story engine, family / history event data, and AI inference, which serve as educational and entertainment elements, allowing users to experience alternative history scenarios on a family-by-family basis. The key can be seen as implementing a time-series face synthesis / inference model into an avatar and an automatic avatar generation logic by age group.

[0219] In addition, the metaverse service device (3120) according to an embodiment of the present invention can further perform operations related to (3) a chatbot / avatar of a deceased person (past person), (4) a group photo → 3D snapshot space, (5) a multi-user simultaneous access / memorial / commemorative event, (6) a memory experience related to five senses expanded memory experience, (7) memory correction through AI sentiment analysis, and (8) a permanent preservation system. (3) The operation related to a chatbot / avatar of a deceased person (past person) involves training an AI model with past data such as text, voice, and video to provide a function for a deceased person avatar to converse with a user in the metaverse. The key here is to extract the speech patterns and voice characteristics of a specific person and reproduce them as a conversational agent. (4) Through the operation related to a group photo → 3D snapshot space, individuals in a single group photo can be automatically segmented and recognized, and the background can be restored in 3D to reproduce the event at that time within the metaverse. The key here is to convert a 2D group photo into a 3D space, connect it to individual nodes (face recognition), and create virtual events. (5) Regarding multi-user simultaneous access and memorial / commemorative events, multiple people can log in at the same time and interact in real time (voice, chat, avatar actions) for events such as weddings, memorial days, and family parties. Through this, a metaverse event system can be implemented where multiple users share, edit, and play node-based memory content (photos, videos, BGM).

[0220] The metaverse service device (3120) according to an embodiment of the present invention (6) can perform operations for a memory experience that expands the five senses. To this end, a memory restoration system based on scents and food recipes can be configured, and a virtual hug linked to tactile (glove) sensations can be made possible. Regarding the memory restoration system based on scents and food recipes, the deceased's favorite food recipes and specific scent information can be converted into data. In the metaverse space, recipe and food-related information can be visualized, and VR / AR taste and smell simulations can be performed. Regarding the virtual hug linked to tactile (glove) sensations, when meeting a family avatar in the metaverse and performing a virtual hug, a tactile feedback device can be used to actually feel the sensation of the hug. Of course, such feedback can also be controlled by a vibration motor installed in the user terminal device (3100). To this end, a tactile feedback suit / glove, real-time motion capture, and avatar synchronization technology within the metaverse can be implemented. Through this, physical sensations can be reproduced to some extent even with family and acquaintances far away, maximizing the sentiment of sharing memories.

[0221] The metaverse service device (3120) can further perform (7) operations related to memory correction through AI sentiment analysis. To this end, it can perform memory modification and correction simulation operations. The AI ​​recognizes that human memory is subjective and can be unstable, and re-edits the color tone, background music (BGM), and atmosphere of photos and videos according to the user's emotional state. For example, when the user is depressed, the memory video is edited to be brightly corrected, and when the user is happy, it is edited with calm background music to enable the reproduction of memories. Through this, a slightly philosophical and psychological healing concept can be applied, processing real-life memories for the purpose of emotional healing. Collaborative editing and remixing operations can be performed for memory correction operations. Multiple family members can simultaneously re-edit past photos and videos (e.g., filters, subtitles, music, etc.) and create a final version as a new memory to post on the metaverse. To this end, collaborative editing tools and real-time reflection of emotions and opinions (e.g., voting, etc.) can be implemented.

[0222] Finally, the metaverse service device (3120) can perform operations related to the (8) permanent preservation system. To ensure that memory data is not lost even if external hacking, errors, or data center fires occur, an IPFS (Distributed File System) or encryption distributed storage and self-recovery protocol can be implemented on multiple nodes. Through this, the concept of a digital vault, which aims for long-term preservation of 100 or 200 years, can be implemented in the metaverse. In addition, for automatic notifications to descendants and continuous donations, even if the account is neglected as generations change, the AI ​​can periodically send family legacy backup guidance to descendants. For example, when the great-grandchild generation appears, automatic email / metaverse message sending is provided. Through this, a truly permanent management structure can be implemented.

[0223] Next, we would like to look further into how the metaverse service device (3120) can operate as an automatic digital inheritance management and dispute mediation system. It can manage a user's digital content (e.g., photos, videos, social media records, NFTs, etc.) as a digital inheritance by unifying it on one platform (web / app / metaverse), allow the user to set a digital will during their lifetime, or allow an heir to inherit the assets without dispute after the user's death through official procedures (e.g., death certificate, etc.), and, in the event of a dispute, allow it to be processed quickly and transparently through a centralized / decentralized Dispute Resolution Module.

[0224] For the above operations, the metaverse service device (3120) can perform operations such as user pre-registration and death certification triggers in relation to digital will and death certification linkage operations. The former means setting whether to use the Digital Will function when signing up for a web / app / metaverse platform, and specifically specifying, for example, to whom and under what conditions one's metaverse avatar, photos, videos, and NFT assets will be inherited upon death, and storing this information (will) in an encrypted form on a blockchain or a separate secure DB. The latter, namely the death certification trigger operation, means that the platform can automatically assign a death status flag to the corresponding user account when a death certificate or death report is received by linking APIs with the government (administrative agency), hospital, insurance company, etc. The automatic processor can operate in a sequence such as activating the digital estate inheritance module when death is confirmed.

[0225] Additionally, the metaverse service device (3120) performs inheritance distribution and automatic access rights transfer operations for the above operations, and to this end, it may perform operations such as conditional smart contracts, multi-signatures, or multiple approvals and rights transfer. Regarding conditional smart contracts, a smart contract is created in advance according to a will, and when death is confirmed, the contract transfers ownership and access rights of property (e.g., digital assets) to specific heirs. For example, detailed settings may be possible, such as disclosing some of my social media records only to my spouse, or sharing sensitive information in photos only with my children. Regarding multi-signatures or multiple approvals, when there are multiple family members (or legal heirs), it can be processed with a multiple approval structure (e.g., one wife, two children, one legal representative, etc.) so that final access is allowed only upon consent. Regarding the transfer of authority, ownership and management rights of a deceased person's avatar in the metaverse platform are automatically transferred to the heir, and the heir can choose to maintain, delete, or make public the corresponding avatar and memory space.

[0226] The metaverse service device (3120) can execute a dispute mediation module to perform operations such as automatic arbitration (or stages) when a dispute occurs, AI-based analysis of relationships and supporting documents, and procedures for agreement and objection. In the automatic arbitration stage when a dispute occurs, if a dispute arises between heirs A and B regarding "who should actually have more rights," the platform can initiate an arbitration process. Here, on-chain arbitration may mean that a pre-agreed group of arbitrators (Arbitration Committee) or the platform's own algorithm comprehensively reviews (1) official documents, (2) will records left by the user, etc., and draws a conclusion. Regarding the AI-based analysis of relationships and supporting documents, for example, the AI ​​can automatically identify genealogical records, family registers, marriage and divorce documents, etc., to determine whether A is the spouse and whether B is not the legal representative, and record the final conclusion on the blockchain to transparently record who inherited when and through what procedure. Regarding settlement and objection procedures, the system can be configured to allow for escalation to actual court proceedings (notarization) if the outcome of the dispute is unsatisfactory. However, utilizing the arbitration procedures provided by the platform enables faster resolution and reduces the likelihood of resorting to external litigation.

[0227] In addition, the metaverse service device (3120) can perform operations related to node (e.g., digital assets) encryption and access levels to enhance privacy and security, as well as operations related to compliance with international regulations such as the General Data Protection Regulation (GDPR) and the Personal Information Protection Act (CCPA). Regarding the former, the deceased's data (photos, messages, personal records) may be encrypted and stored in advance, and even heirs may be required to undergo separate authentication to view sensitive information. Compliance with personal information protection and international regulations such as the GDPR and CCPA may be required. Data is disclosed to heirs only within the scope agreed upon by the deceased during their lifetime, and the scope of disclosure / non-disclosure may be provided according to the settings of the will.

[0228] The third-party device (3130) may include various types of devices for providing a family data-based metaverse service according to an embodiment of the present invention by linking with the metaverse service device (3120) of FIG. 9. Typically, the third-party device (3130) may include a server or computer of a developer that develops a program according to an embodiment of the present invention and installs it on the metaverse service device (3120) of FIG. 9, and may also include a computer of an administrator that manages the metaverse service. Above all, the third-party device (3130) may include an SNS server that provides SNS. In the case of an SNS server, it can be of great help in supplementing family data. The third-party device (3130) may link with the metaverse service device (3120) via an API (application programming interface) to provide the family data of the user in real time or periodically at time intervals based on the user's account information.

[0229] In addition to the above, the user terminal device (3100), communication network (3110), metaverse service device (3120), and third-party device (3130) of FIG. 9 can perform various operations, and since the relevant details will continue to be covered later, the details will be replaced by those details.

[0230] FIG. 10 is a block diagram illustrating the detailed structure of the metaverse service device of FIG. 9, and FIG. 11 is a block diagram illustrating the detailed structure of the metaverse service unit of FIG. 10.

[0231] As illustrated in FIG. 10, a metaverse service device (3120) according to an embodiment of the present invention includes some or all of a communication interface unit (3200), a control unit (3210), a metaverse service unit (3220), and a storage unit (3230).

[0232] Here, "including some or all" means that the metaverse service device (3120) may be configured with some components, such as the storage unit (3230), omitted, or that some components, such as the metaverse service unit (3220), may be configured by integrating them into other components, such as the control unit (3210). To facilitate a sufficient understanding of the invention, it is described as including all components.

[0233] The communication interface unit (3200) communicates with the user terminal device (3100) and the third-party device (3130) respectively via the communication network (3110) of FIG. 9. During the process of performing communication, the communication interface unit (3200) performs operations such as modulation / demodulation, encoding / decoding, muxing / demuxing, and scaling to convert resolution, which are obvious to those skilled in the art, so further explanation is omitted.

[0234] The communication interface unit (3200) can provide metaverse services under the control of the control unit (3210) when there is a service request from the user terminal device (3100) of FIG. 9. Of course, the communication interface unit (3200) according to the embodiment of the present invention provides services for creating a family data-based metaverse avatar and implementing a memory place. To this end, it can collect family data for creating an avatar and implementing a memory place from the user terminal device (3100) and provide it to the control unit (3210). Of course, when the creation of an avatar and the implementation of a memory place are completed, the communication interface unit (3200) can perform various operations, such as modifying data related to family relationships during the process of using the service or receiving and processing additional family data provided by the user terminal device (3100). Of course, in addition to this, the communication interface unit (3200) can also participate in digital legacy-related operations or user dispute resolution operations as previously described under the control of the control unit (3210).

[0235] The communication interface unit (3200) may perform operations such as supplementing family data through API integration with a third-party device (3130). The communication interface unit (3200) may receive family data of family members, such as that provided periodically by the third-party device (3130) via an SNS server, and provide it to the control unit (3210) for data analysis.

[0236] The control unit (3210) may include a processor such as a CPU, MPU, or GPU, and may further include memory such as RAM. It is also possible for the processor and memory to be configured as a single chip, such as an IC chip. The control unit (3210) can perform overall control operations of the communication interface unit (3200), metaverse service unit (3220), and storage unit (3230) of FIG. 10. When there is a service request from the user terminal device (3100) of FIG. 9, the control unit (3210) can, of course, proceed with procedures for membership registration, etc., during the initial service period, and can perform operations to collect family data during this process or thereafter. For example, when there is a service request from the user terminal device (3100), the control unit (3210) can execute a graphic (GUI) program of the metaverse service unit (3220) to provide a visualized screen such as a graphic screen. Of course, such a screen can be a service screen with a designated layout. For example, it is possible to input family member information so that the relationships between family members can be known, or to receive data such as photos or videos of the family tree. For example, the layout screen may allow input of family member information by year, or may allow input of birth and death information of family members. Accordingly, the control unit (3210) may also provide a service screen in the form of a family tree generated and provided by the metaverse service unit (3220).

[0237] In addition, the control unit (3210) can be involved in operations such as creating family data-based avatars, 3D reproduction of places of memory, expanding worldviews based on security and consent, providing emotion-centered immersive experiences, managing digital family legacy, and serving as a global social metaverse connection hub, although the actual specific operations are executed by the metaverse service unit (3220). Furthermore, it can be involved in operations such as the automatic inheritance management of digital legacy or dispute mediation as previously described. For example, for creating avatars or 3D reproduction of places of memory, an AI model of an artificial intelligence program can be installed in the metaverse service unit (3220) to perform learning operations using training data. Of course, a generative AI model may be preferable for creating avatars, and accordingly, if photo or video data of family members is input, an avatar can be created based on this. Of course, in the case of avatars, the form of the avatar can be seen as changing over time for the same person. It can be seen as changing the avatar to reflect the times, such as wrinkles or hairstyles. In addition, regarding memory locations, the background in the photo is analyzed to create the memory location, and it is entirely possible to create the memory location more accurately based on pre-learned training data to create a more precise memory location. Of course, the specific avatar creation and memory location creation described above are performed by the metaverse service unit (3220), but the control unit (3210) may be involved in these operations. Furthermore, since the operation of the control unit (3210) may be performed by copying or storing a program for the above operations in the form of an IC chip in the internal memory and executing it, the embodiment of the present invention will not be specifically limited to any one form.

[0238] The metaverse service unit (3220) can perform avatar creation operations based on family data. To this end, the metaverse service unit (3220) can create 3D avatars that reflect the appearance, voice, personality, and behavioral patterns of family members. Additionally, the metaverse service unit (3220) can implement natural behavior and conversation patterns that reflect the characteristics of family members through AI-based learning. The generated avatars may differ in appearance by era (or year) for the same person, and the flow of the times can be reflected by changing features such as wrinkles or hairstyles. Of course, for wrinkles or hairstyles, family data such as photos or videos can be utilized. For instance, the appearance of family members in the 1970s and the 1980s are different, and this can be reflected in the avatar. Furthermore, the metaverse service unit (3220) can recreate 3D memories of places. The metaverse service unit (3220) can recreate family memories of places in a 3D virtual space based on photos, videos, GPS data, etc. In other words, the virtual space corresponds to the background in which the avatar operates, and can support the use of AI photogrammetry or restoration technology to supplement incomplete data and set custom environments (e.g., weather, time zone, etc.).

[0239] Of course, in addition to this, the metaverse service unit (3220) can perform various other operations. For these operations, the metaverse service unit (3220) includes some or all of the data linkage and synchronization unit (3300), avatar creation unit (3310), memory place reproduction unit (3320), security and consent-based extension unit (3330), emotion sensing and interaction unit (3340), digital legacy management unit (3350), and global social extension unit (3360), as shown in FIG. 11. Of course, the metaverse service unit (3220) may also include a manager in the form of software (3 / W) that controls the above components in conjunction with the control unit (3210) of FIG. 10. Here, "including some or all" is not significantly different from the meaning above, and the above components may be composed of hardware (H / W) modules, software (3 / W) modules, or a combination thereof.

[0240] The data linkage and synchronization unit (3300) is equipped with and executes a data linkage and synchronization module, thereby synchronizing data between a user web or app program and a metaverse platform in real time and enabling integrated management of data in various formats, such as family tree data, photos, and videos. The avatar creation unit (3310) executes an avatar creation module and utilizes AI deep learning technology to create 3D avatars that reflect the appearance, voice, and personality of family members. It can also implement natural interactions through the learning of behavior and conversation patterns. The memory location reproduction unit (3320) executes a memory location reproduction module and utilizes AI-based photogrammetry or photo enhancement technology to restore photos, videos, and GPS data into a 3D environment, and can provide a personalized virtual space where user-specified environments (e.g., lighting, weather) can be set. Here, GPS data may be used, for example, to change the frame rate of a video. For example, a typical example is switching the screen when the location of a family member changes based on GPS data. The security and consent-based extension unit (3330) executes the security and consent-based extension module and can operate as a security system that allows access only to individuals designated by the user. Personal data can be safely managed through invitation-based connections and private space setting functions. The emotion sensing and interaction unit (3330) executes the emotion sensing and interaction module, analyzes user emotions in real time to personalize avatars and environments, and can provide emotional immersion through AI conversation and behavior generation technology. The digital heritage management unit (3340) executes the digital heritage management module and can safely store family data and memories and manage them in a format that can be transmitted between generations. The global social extension unit (3340) executes the global social extension module and can support data linkage and real-time interaction with global users. Cooperation between various cultures can be promoted through translation and interpretation AI and global event functions.

[0241] In addition, the data linkage and synchronization unit (3300), avatar creation unit (3310), memory place reproduction unit (3320), security and consent-based extension unit (3330), emotion sensing and interaction unit (3340), digital legacy management unit (3350), and global social extension unit (3360) as shown in FIG. 11, which constitute the metaverse service unit (3220), can perform various operations, and related details will be discussed further with reference to FIG. 12. However, the parts related to automatic digital legacy inheritance management or dispute mediation will be replaced by the aforementioned details.

[0242] The storage unit (3230) can temporarily store various types of information or data processed under the control of the control unit (3210). When family data such as photos, videos, and genealogies is provided from the user terminal device (3100) of FIG. 9, for example, to form a family tree, the storage unit (3230) can temporarily store this information and then provide it to the metaverse service unit (3220) for data processing.

[0243] In addition to the above, the communication interface unit (3200), control unit (3210), meta service unit (3220), and storage unit (3230) of FIG. 10 can perform various operations, and since other details have been sufficiently explained above, the details will be replaced by those details.

[0244] According to an embodiment of the present invention, the communication interface unit (3200), control unit (3210), meta service unit (3220), and storage unit (3230) of FIG. 10 are composed of hardware modules that are physically separated from each other, but each module may store software for performing the above operations internally and execute it. However, since the software is a set of software modules and each module can be formed as hardware, the configuration will not be specifically limited to software or hardware. For example, the storage unit (3230) may be storage or memory, which are hardware. However, since it is also possible to store information in a software-based repository, the above content will not be specifically limited.

[0245] Meanwhile, as another embodiment of the present invention, the control unit (3210) may include a CPU and memory and may be formed as a single chip. The CPU includes a control circuit, an arithmetic logic unit (ALU), an instruction interpretation unit, and a registry, and the memory may include RAM.

[0246] The control circuit can perform control operations, the operation unit can perform operations on binary bit information, and the instruction interpretation unit can perform operations to convert high-level language into machine language and machine language into high-level language, including an interpreter or compiler, and the registry can be involved in software data storage. According to the above configuration, for example, at the beginning of the operation of the metaverse service device (3120), the data operation processing speed can be rapidly increased by copying a program stored in the metaverse service unit (3220), loading it into memory, i.e., RAM, and then executing it. In the case of a deep learning model, it may be loaded into GPU memory instead of RAM and executed by using the GPU to accelerate the execution speed.

[0247] FIG. 12 is a flowchart illustrating a metaverse service process according to an embodiment of the present invention.

[0248] For convenience of explanation, referring to FIG. 12 together with FIG. 9, the metaverse service device (3120) according to an embodiment of the present invention can, for example, execute the module of the data linkage and synchronization unit (3300) of FIG. 11 to collect family data through communication with a user terminal device (3100) and, if there is family data stored in the DB (3120a), perform a retrieval operation (S3400). Retrieving family data may correspond to a basic data collection step for providing user data to the metaverse platform. Accordingly, photos, videos, and GPS location data can be collected and standardized in real time through API and cloud technology. This standardized data can be transmitted when creating a 3D avatar to configure the appearance and voice of the avatar.

[0249] In addition, data can be provided to recreate nostalgic places, serving as foundational material for reconstructing virtual spaces. Furthermore, through data updates and storage, collected data can be continuously synchronized and stored on the platform.

[0250] Additionally, the metaverse service device (3120) can perform a 3D avatar generation operation by executing the module of the avatar generation unit (3310) of FIG. 11 (S3410). The metaverse service device (3120) can generate realistic avatars that reflect the facial, voice, and behavioral characteristics of family members using AI deep learning and GAN (Generative Adversarial Networks) technology. While AI deep learning is used for the analysis of family data, GAN technology can be used for avatar generation as a generative AI. The metaverse service device (3120) can create avatars capable of emotional interaction through age change simulation and behavior prediction algorithms.

[0251] The metaverse service device (3120) can be connected to emotion sensing and interaction to set the avatar to react according to the user's emotional state. Additionally, the avatar can be active and interact within the recreated space in the recreation of a memory place. Data of the created avatar can be updated and saved through data updating and saving. The metaverse service device (3120) can be involved in these operations.

[0252] The metaverse service device (3120) can recreate a high-resolution 11D space by executing the module of the memory place reproduction unit (3320) of FIG. 11 and analyzing the user's photos, videos, and GPS data through AI-based Photogrammetry and NeRF (neural Radiance Fields) technology. Here, NeRF may refer to a technology that generates a new viewpoint image from multiple viewpoint images of a static object. In other words, NeRF is a technology that creates 3D from, for example, a captured 2D image, and can use deep learning technology. It is a technology that performs 3D modeling from a 2D photographic image through a neural network learning method, and predicts the color, density, shape, and texture of a specific point when the viewpoint of the resulting 3D image is moved, that is, when the view is changed, thereby enabling 3D modeling. It can also be seen as performing rendering operations based on photos. For example, the memory place reproduction unit (3320) can maximize immersion through physically based rendering (PBR) and an environment simulation engine. The recreation of memorable places can be linked to the creation of 3D avatars, allowing the avatars to interact in a virtual space. Additionally, access to the recreated space can be restricted to authorized users through security and approval systems, and data from the recreated place can be synchronized in real time through data updates and storage.

[0253] The metaverse service device (3120) can securely manage user data through blockchain-based access control and smart contracts via the security and permission-based extension unit (3330) of FIG. 11, and allow access only to authorized users. The security and permission-based extension unit (3330) can manage access rights for the reproduction of memory places, record and manage data access logs in connection with data updates and storage, and protect interaction data of emotion sensing and interaction to ensure privacy. In other words, it can perform operations related to this.

[0254] The metaverse service device (3120) can analyze the user's facial expressions, voice, and behavior data in real time through Emotion AI and Multimodal Sensor Fusion technology via the emotion sensing and interaction unit (3340) of FIG. 11.

[0255] Based on the analysis results, the reactions of the avatar and the virtual environment can be personalized. Multimodal fusion refers to a technology that integrates various forms of data to make better analyses and decisions. The emotion sensing and interaction unit (3340) interacts with the creation of a 3D avatar and can adjust the avatar's reactions to match the user's emotions. It can also continuously improve by transmitting emotion analysis data for data updates and storage, and can enhance the user experience of interacting with the avatar in the recreation of a memorable place.

[0256] The metaverse service device (3120) can support seamless interaction between multilingual users by utilizing translation and interpretation AI through the global novel extension (3360) of FIG. 11. It can also activate connections between users of various cultures and backgrounds through global events and programs. The global novel extension (3360) can provide an emotional experience between global users in combination with 3D avatar creation, support various languages ​​and emotional expressions in real time along with emotion sensing and interaction, and store and analyze multilingual interaction and global event data through data updates and storage.

[0257] The metaverse service device (3120) can perform data update and storage operations by executing a module mounted on a data update and storage unit, such as the data linkage and synchronization unit (3300) of FIG. 11, and can store and synchronize all data generated within the platform (e.g., user interaction, emotion analysis, avatar activity, etc.) in real time. The platform performance can be continuously improved by training a machine learning model with data. For example, the data linkage and synchronization unit (3300) of FIG. 11 can support all processes within the platform based on data collected from family data retrieval, and can manage 3D avatar creation, recreation of memory places, emotion sensing, and interaction data in an integrated manner. In addition, access rights and security of the data can be maintained through a security and approval system.

[0258] In addition to the above, the metaverse service device (3120) according to the embodiment of the present invention can perform various operations, and other detailed information has been sufficiently explained above, so it will be replaced by that information.

[0259] Figure 13 is a flowchart showing the operation process of the metaverse service device of Figure 9.

[0260] For convenience of explanation, referring to FIG. 13 together with FIG. 9, a metaverse service device (3120) according to an embodiment of the present invention can communicate with a user terminal device (3100) of a user who intends to use a family relationship-centered metaverse service (S3500). The user terminal device (3100) can access the metaverse service device (3120) of FIG. 9 through a web or app service and provide family data after signing up. Of course, here, the family data may include various types of data. Information such as the birth dates of family members or detailed information such as age may be provided. In addition, the family data may provide photos, videos, and genealogical data related to family members. Of course, the family data may include various data related to the activities of family members provided by an SNS server, etc., that constitutes the third-party device (3130) of FIG. 9.

[0261] Additionally, the metaverse service device (3120) can apply family data provided through communication with the user terminal device (3100) to an artificial intelligence (AI) program to create an avatar that reflects the characteristics of family members, recreate a memory location associated with the created avatar, and provide a metaverse service to the user terminal device (3100) where the created avatar is active in the recreated memory location (S3510). For example, the metaverse service device (3120) may include a deep learning-based AI model for analyzing family data, and may create an avatar based on the analysis results from the deep learning-based AI model, and may execute a generative AI model for this purpose. In this way, the output of the deep learning-based AI model (3 or the first AI model) can become the input of the generative AI model (hereinafter, the second AI model), and in this process, it is entirely possible to convert it into the form of a prompt value to facilitate input to the generative AI model.

[0262] For example, the metaverse service device (3120) provides a service screen with a designated layout structure, and it is entirely possible to generate a family tree of family members by year and provide family data through it. Of course, the user of the user terminal device (3100) can set the names or relationships of family members by year, but if, for example, photos related to a family tree are provided, it is entirely possible to automatically generate a family tree or a family tree by year by reading the photos using OCR (Optical Character Recognition) technology and analyzing the OCR-based text using a Large Language Model (LLM). Of course, rather than generating a family tree, the metaverse service device (3120) can primarily perform operations to generate avatars of family members by era based on the family data provided through the tree, and to recreate nostalgic places where one can see the social conditions of that time. For example, the appearance of the father in the 1970s and the father in the 1980s may be different. Of course, the complete appearance of the family members at that time can be directly reflected in the avatars and implemented on the metaverse service screen, and the nostalgic locations serving as the background can also be recreated exactly as they were at the time. While nostalgic locations can be recreated using generative AI models based on their appearance in photographs, it is entirely possible to recreate them based on pre-trained data related to those locations in the past, even in cases where data is insufficient.

[0263] In addition to the above, the metaverse service device (3120) of FIG. 9 can perform various operations, and other details have been sufficiently explained above, so they will be used instead.

[0264] Although all components constituting an embodiment of the present invention have been described as being combined or operating as a single unit, the present invention is not necessarily limited to such an embodiment. That is, within the scope of the purpose of the present invention, all components may be selectively combined and operated in one or more ways. Furthermore, while all components may each be implemented as a single independent piece of hardware, they may also be implemented as a computer program having a program module that performs some or all of the combined functions on one or more pieces of hardware by selectively combining some or all of the components. The codes and code segments constituting the computer program can be easily inferred by those skilled in the art of the present invention. An embodiment of the present invention may be implemented by storing such a computer program on a non-transitory computer-readable medium, reading it, and executing it by a computer.

[0265] Here, a non-transient readable recording medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specifically, the programs described above may be stored and provided on non-transient readable recording media such as CDs, DVDs, hard disks, Blu-ray discs, USBs, memory cards, and ROMs.

[0266] Although embodiments of the present invention have been described above with reference to the attached drawings, those skilled in the art will understand that the present invention may be implemented in other specific forms without changing the technical concept or essential features thereof. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive.

[0267] Figure 14 is a conceptual diagram illustrating an exemplary digital self-curation system.

[0268] Referring to FIG. 14, a digital self-curation system (4010) according to the present invention may include a server (4100) that acquires and stores emotional input data from a receiving device, an ethical judgment device (4200) that evaluates the risk level of the emotional input data and performs emotional refinement, and a self-generation device (4300) that generates emotional output data corresponding to the emotional input data.

[0269] The ethical judgment device (4200) performs risk calculation and refinement decisions, and the final generation of emotion output data can be performed by the self-generation device (4300). The ethical judgment device (4200) can generate at least some of the risk, sensitivity, emotion output data generation instructions, and sensitivity weights and transmit them to the self-generation device (4300).

[0270] The receiving device can collect language input data including at least some of the operator's voice, handwriting, and text, and emotion input data including at least some of the operator's behavior, facial expressions, gaze, posture, gestures, tone of voice, intonation, neural signals, biosignals, environmental information, and respiration.

[0271] The server (4100) can acquire emotion input data from a receiving device, store emotion input data and emotion output data in a database, and transmit emotion input data to an ethics judgment device (4200). Additionally, it can store self-generation basic data, including at least some of the user's conversation records, diaries, essays, memos, videos, audio, and handwritten notes, in a database and provide it to a self-generation device.

[0272] When the ethics judgment device (4200) obtains emotion input data from the server (4100), it calculates a risk level based on at least some of the social ethics standards and community ethics standards, and if the risk level is judged to be greater than or equal to a preset standard risk level, it performs emotion refinement on the emotion input data. At this time, the social ethics standards are generated based on the ethics of the social norms to which the agent and / or user belong, and the community ethics standards can be generated based on the ethics unique to the community to which the agent and user belong together. For example, social ethics standards may be established based on the norms of Korean society; if the agent and user belong to the same family community, community ethics standards may be established based on the ethics standards shared by the family community; and if the agent and user belong to the same workplace community, community ethics standards may be established based on the ethics standards shared by the workplace community. Community ethics standards may be generated individually depending on the relationship between the user and the agent. To generate these ethics judgment standards, the ethics judgment device (4200) may include an ethics judgment model that includes at least some of a pre-trained artificial intelligence, a machine learning model, a deep learning network, and a statistical-based prediction model.

[0273] The risk level of the emotion input data is calculated based on ethical judgment criteria, and emotion refinement can be performed if the risk level is determined to be higher than a preset standard risk level. In this case, the standard risk level may include a first standard risk level and a second standard risk level having a value higher than the first standard risk level; if the risk level is determined to be between the first and second risk levels, expressions with a high risk level can be modified into refined expressions with a similar meaning, and if the risk level is determined to be above the second risk level, the corresponding emotion input data can be removed.

[0274] The self-generation device (4300) generates emotion output data including at least some of the corresponding language output data and non-language output data based on emotion input data from which emotion refinement has been completed from the ethics judgment device (4200). To generate emotion output data, a digital self-generation model may be included that includes at least some of the pre-trained artificial intelligence, machine learning model, deep learning network, and statistical-based prediction model. The digital self-generation model may be trained based on an emotion learning data pair that includes the emotion label data generated in correspondence with the emotion learning data, through a pre-trained emotion inference model that obtains self-generation basic data from the server (4100), generates emotion learning data including at least some of the user's textual language learning data and non-language learning data including at least some of the user's behavior, facial expression, gaze, posture, gesture, tone of voice, intonation, and breathing based on the same, and generates label data in which the emotion corresponding to the emotion learning data is labeled.

[0275] FIG. 15 is a conceptual diagram exemplifying the configuration of an ethical judgment device, and FIG. 17 is a schematic diagram exemplifying the configuration of a self-generation device.

[0276] Referring to FIG. 15, the ethics judgment device (4200) may include a risk assessment processor (4201), RAM (Random Access memory) (4202), an ethics norm database (4203), and a communication unit (4204). The self generation device (4300) may include a processor (4301), RAM (Random Access memory) (4302), storage (4303), and a communication unit (4304). The ethical judgment device (4200) and the self-generation device (4300) can be implemented as various types of devices, for example, a laptop computer, a mobile phone, a smartphone, a tablet PC, a mobile internet device (MID), a personal digital assistant (PDA), an enterprise digital assistant (EDA), a digital still camera, a digital video camera, a portable multimedia player (PMP), a personal navigation device or portable navigation device (PND), a handheld game console, an e-book, or a smart device. The smart device can be implemented as a smart watch, a smart band, or a smart ring.

[0277] The risk assessment processor (4201) can control the overall operation of the ethics judgment device (4200), and the processor (4301) can control the overall operation of the self-generation device (4300). The risk assessment processor (4201) and the processor (4301) may include a single processor core or a CPU (Central Processing Unit) that includes multiple processor cores. The ethics judgment device (4200) may include one or more risk assessment processors (4201), and the self-generation device (4300) may include one or more processors (4301).

[0278] For example, the risk assessment processor (4201) and the processor (4301) may include a microprocessor, a central processing unit, a processor core, a multi-core processor, a multiprocessor, an ASIC (Application-Specific Integrated Circuit), and an FPGA (Field Programmable Gate Array).

[0279] The risk assessment processor (4201) can process or execute programs, data, or instructions stored in the ethical code database (4203). For example, the risk assessment processor (4201) can update the database by executing programs stored in the ethical code database (4203).

[0280] The processor (4301) can process or execute programs, data, or instructions stored in storage (4303). For example, the processor (4301) can update the database by executing programs stored in storage (4303).

[0281] RAM (4202) and RAM (4302) may temporarily store programs, data, or instructions. For example, programs and / or data stored in the ethics code database (4203) may be temporarily stored in RAM (4202) under the control of the risk assessment processor (4201) or boot code. For example, RAM (4202) includes DRAM (Dynamic RAM), SRAM (Static RAM), SDRAM (Synchronous DRAM), etc.

[0282] The ethical code database (4203) and storage (4303) are storage locations for storing data, and can store an OS (Operating System), various programs, and various data. The ethical code database (4203) includes ROM (Read Only Memory), flash memory, PRAM (Phase-change RAM), MRAM (Magnetic RAM), RRAM (Resistive RAM), FRAM (Ferroelectric RAM), etc. In an embodiment, the ethical code database (4203) can be implemented as an HDD (Hard Disk Drive), SSD (Solid State Drive), etc.

[0283] The communication unit (4204) can transmit and / or receive data from the ethics judgment device (4200). The communication unit (4304) can transmit and / or receive data from the self-generation device (4300). For example, the communication unit (4204) and the communication unit (4304) may be configured to communicate with various types of external devices according to various types of communication methods. The communication unit (4204) may include at least one of a network chip, a Wi-Fi chip, a Bluetooth chip, a wireless communication chip, and an NFC chip. For example, each of the Wi-Fi chip and the Bluetooth chip can perform communication in a Wi-Fi method and a Bluetooth method, respectively. When using a Wi-Fi chip or a Bluetooth chip, various connection information such as SSID and session key may be transmitted and received first, and then various information may be transmitted and received after establishing a communication connection using this information. A wireless communication chip refers to a chip that performs communication according to various communication standards such as IEEE, Zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), and 5G (5th Generation). A wireless communication chip according to 5G communication standards can use not only frequency bands such as 3.5GHz (Below 6GHz) but also millimeter wave (mmWave), i.e., ultra-high frequency bands (Above 6GHz), such as 26, 28, 38, 39, and 60 GHz. An NFC chip refers to a chip that operates using the NFC (Near Field Communication) method, which utilizes the 13.56MHz band among various RF-ID frequency bands such as 135kHz, 13.56MHz, 433MHz, 860~960MHz, and 2.45GHz.

[0284] Accordingly, the ethics judgment device (4200) and self-generation device (4300) of the present disclosure can perform operations by means of the above configurations, temporarily store data or commands, or perform data transmission and / or reception with other management servers.

[0285] The ethics judgment device (4200) and the self generation device (4300) may include at least one of a cloud, a virtual server, IaaS, PaaS, SaaS, an edge, a distributed computing, a local processing unit, and a server, and the risk calculation, emotion refinement, and emotion output data generation may be executed in a distributed manner on at least one of a cloud, a virtual server, IaaS, PaaS, SaaS, an edge, a distributed computing, a local processing unit, and a server. Hereinafter, the description will be made with reference to FIGS. 14 to 15 and FIG. 17.

[0286] FIG. 16 discloses a flowchart showing the operation method of an ethical judgment device.

[0287] Referring to FIG. 16, the ethics judgment device (4200) obtains (S4100) from a server (4100) language input data including at least some of the operator's voice, handwritten notes, and text, and emotion input data including at least some of the operator's behavior, facial expressions, gaze, posture, gestures, tone of voice, intonation, breathing, neural signals, biosignals, and environmental information.

[0288] The ethical judgment device (4200) can cause the risk assessment processor (4201) to generate a risk level (S4200) based on at least some of the social norm taboo expressions and community taboo expressions.

[0289] The above risk level may be a continuous value from 0 to 1 as an indicator of the possibility that the emotional input data may violate ethical standards, including social norm taboo expressions and community taboo expressions.

[0290] At this time, the risk assessment processor (4201) may include an ethical judgment model that includes at least some of the artificial intelligence, machine learning model, deep learning network, and statistical-based prediction model that have been pre-trained for risk assessment. The ethical judgment model is characterized by being trained based on a pair of ethical judgment learning data that includes: language learning data that includes at least some of the voice, handwriting, and text of the operator that have been pre-stored and obtained from the server (4100); emotion learning data that includes at least some of the non-verbal learning data that includes at least some of the operator's behavior, facial expression, gaze, posture, gesture, tone of voice, intonation, breathing, neural signals, biosignals, and environmental information; emotion output data that includes emotion type data and emotion intensity data corresponding to the emotion learning data; and risk learning output data that includes community norm risk and social norm risk, which are moral and ethical standards based on social norms, extracted based on the emotion output data.

[0291] When the risk assessment processor (4201) obtains the emotional input data of the operator from the server (4100), it causes the ethical judgment model to generate, based on this, social norm taboo expressions that are contrary to the moral and ethical standards shared by the society to which the user belongs and social norm risk judgment criteria corresponding to said social norm taboo expressions, generate community taboo expressions that are contrary to the moral and ethical standards shared by the community to which the user and the operator belong and community norm risk judgment criteria corresponding to said community taboo expressions, and generate a risk level (S4200) based on said social norm risk judgment criteria and said community norm risk judgment criteria.

[0292] At this time, the sensitivity may be a continuous value of 0 or more and 1 or less as a weight that reflects the characteristics of the operator.

[0293] In this context, the criteria for assessing social norm risk may include domestic and international codes of ethics and social norms. For instance, discrimination or defamation based on race, nationality, gender, or religion may generate a high risk level according to these criteria. The criteria for assessing community norm risk may include norms specific to the community to which both the user and the agent belong. For instance, if the user and the agent belong to the same family community, the risk level could be generated based on factors such as whether honorifics are used toward a specific individual within the family relationship.

[0294] The risk assessment processor (4201) can refine the emotion input data (S4300) based on the generated risk level. At this time, if the risk level is determined to be greater than a preset first standard risk level or less than a preset second standard risk level which is greater than the first standard risk level, it is determined to be a sensitive emotion expression and refined into an appropriate expression, and if the risk level is determined to be greater than the second standard risk level, it is determined to be a risk emotion expression and the risk emotion expression can be blocked.

[0295] For each of the multiple agents, the criteria for judging social norm risk and community norm risk, as well as domestic and international codes of ethics, social norms, sensitive expressions specific to the culture to which at least some of the users and agents belong, and at least some of the public institution standards can be stored in the ethics norm database (4203).

[0296] When the risk assessment processor (4201) generates the risk level, it may additionally classify the operator sensitivity based on at least some of the operator's country, age group, cultural background information and relationship with the user, and further reflect the operator sensitivity in the risk level.

[0297] For example, if the operator belongs to a specific race, the risk level could be adjusted by assigning a low sensitivity to certain racist language directed at that race; similarly, if the operator belonging to the same family community is young or is a foreigner, the risk level could be adjusted by assigning a low sensitivity to honorifics. Sensitivity can be set individually for each user and multiple operators, and can be reset or adjusted based on the operator's age or the time of the conversation.

[0298] Additionally, the risk assessment processor (4201) may evaluate the real-time risk based on linguistic response data including at least some of the operator's voice, handwriting, and text from the interaction with the operator, and emotional response data including at least some of the operator's behavior, facial expression, gaze, posture, gesture, tone of voice, intonation, and breathing, and may adjust the sensitivity in real-time based on the rate of change of the real-time risk. For example, in a situation where the operator and the processor exchange jokes about a specific topic, if the risk of the emotional response data gradually increases, the flow of the conversation may be maintained by adjusting the sensitivity to gradually decrease.

[0299] The risk assessment processor (4201) may further include a plurality of interaction learning models learned based on the interaction patterns of each user and each of the plurality of agents, for each user and each of the plurality of agents corresponding to the emotion learning data stored in the ethical norm database (4203).

[0300] The ethical judgment model generates at least some of the sensitivities and risks based on the interaction learning responses of multiple interaction learning models. The multiple interaction learning models construct response sequences based on sentiment learning data and at least some of the generated virtual scenarios, and generate interaction learning responses by iteratively simulating the response sequences. Furthermore, the model can generate an interaction response fit based on the risk and sensitivity of the interaction learning responses, and fine-tune the interaction learning responses based on the interaction learning response fit. Through this process, rich training data pairs can be generated, and the accuracy of the ethical judgment model's sensitivity and risk generation can be improved even with limited sentiment learning data.

[0301] Additionally, when the user status flag changes from living to post-mortem, a weight may be generated for the operator sensitivity, and the weight may be dynamically adjusted to weight the relationship, age, and culture in kinship-targeted utterances in post-mortem mode.

[0302] The risk assessment processor (4201) may determine the emotion learning data as emotion bias data and instruct the server to delete the emotion bias data if a risk level greater than the third standard risk level, which is a preset standard risk level within a preset time for the same agent, is detected in the emotion learning data and the risk level greater than the third standard risk level exceeds a preset standard number of times. For example, if the emotion learning data includes an argument, a risk level greater than the third standard risk level may be repeatedly generated within a specific time, and in such cases, the emotion learning data corresponding to the argument may be deleted to prevent the form of the argument from being learned by the self-generation device.

[0303] The first to third standard risk levels can be set as risk thresholds, for example, to 0.4, 0.7, and 0.85, respectively, and may be dynamically adjusted according to service policies or learning results.

[0304] Additionally, the risk assessment processor (4201) may include a feedback loop structure that acquires emotional response data, which is the response of the agent corresponding to the emotional output data generated by the self-generating device, and reinforces the interaction learning response having a low risk level and automatically corrects the interaction learning response having a high risk level based on the emotional response data and the interaction learning response. Meanwhile, the methods according to the various embodiments of the present invention described above may be implemented in the form of an application or software program that can be installed on an existing electronic device.

[0305] FIG. 18 discloses a flowchart illustrating the operation method of a digital self-curation system.

[0306] Referring to FIG. 18, a server (4100) obtains from a receiving device emotional input data including at least some of the operator's voice, handwriting, and text, and at least some of the operator's behavior, facial expression, gaze, posture, gesture, tone of voice, intonation, neural signals, biosignals, environmental information, and breathing (S5100). At this time, there may be multiple receiving devices, and the server may obtain emotional input data from an external device.

[0307] The ethics judgment device (4200) inputs emotion input data into a pre-learned ethics judgment model to calculate a risk level based on at least some of the social ethics standards and community ethics standards, and performs emotion refinement on the emotion input data when the risk level is judged to be greater than or equal to a pre-set standard risk level (S5200). At this time, the ethics judgment model is characterized by being learned based on risk output data including the risk level generated based on the self-generation basic data including at least some of the user's conversation records, diaries, essays, memos, videos, audio, and handwritten notes stored in the database of the server (4100), generating social norm risk level judgment criteria and community norm risk level judgment criteria from at least some of the social taboo expressions and community taboo expressions. At this time, the social taboo expressions include at least some of the abusive language, verbal abuse, and discriminatory expressions regarding gender, race, origin, and religion defined by social norms, and the community taboo expressions may include at least some of the expressions that are contrary to the moral and ethical standards shared by the community to which the user belongs. The ethical judgment model generates a risk level based on social norm risk judgment criteria and community norm risk judgment criteria, and can refine risk emotion expressions when the risk level is judged to be greater than or equal to a pre-established first standard risk level and less than or equal to a pre-established second standard risk level which is greater than the first standard risk level, and can block risk emotion expressions when the risk level is judged to be greater than or equal to the second standard risk level.

[0308] An emotional input data that has undergone emotional refinement is acquired by a self-generation device (4300), and the emotional input data is input into a digital self-generation model that includes at least some of a pre-learned artificial intelligence, machine learning model, deep learning network, and statistical-based prediction model to generate emotional output data (S5300) that includes at least some of the verbal output data and non-verbal output data corresponding to the input data and non-verbal input data.

[0309] At this time, the ethics judgment device (4200) may further perform risk calculation on the emotion output data generated by the self-generation device (300), and may transmit it to the emotion output device only when it is judged to be below a standard risk level. The self-generation device (4300) and the ethics judgment device (4200) may be composed of multiple devices as described above, and data learning, emotion output data generation, and risk calculation may each be performed in different devices.

[0310] After the self-generating device (4300) generates emotion output data, it may cause an emotion output device (not shown) to output the emotion output data. At this time, the emotion output device may convert the emotion output data into reaction data including at least some of text, voice, and video, and transmit the reaction data to at least some of a display, a hologram device, a speaker robot, a VR device, an AR device, and a mobile device.

[0311] An emotion output device may include at least some of a digital monument, a UI, and a metaverse, and may express non-verbal expressions through voice, avatar, light, or pictures, including at least some of a display, a holographic device, a speaker, a robot, a VR device, an AR device, a haptic device, and a mobile device.

[0312] At this time, the response data may include at least some of text, voice, avatar, light, video, hologram, multimodal signal, and robot motion.

[0313] The digital self-generating device (4300) can acquire response data generated from an operator interacting with emotion output data output by an emotion output device. At this time, the response data may include linguistic response data including at least some of the operator's voice, handwriting, and text corresponding to the emotion output data, and at least some of non-linguistic response data including at least some of the operator's behavior, facial expression, gaze, posture, gesture, tone of voice, intonation, and breathing. The emotion response data may be acquired directly from a receiving device or may be acquired by a server (4100).

[0314] The self-generating device (4300) can generate emotional labeling data corresponding to emotional response data, generate response learning data pairs including emotional response data and emotional labeling data, store emotional output data and emotional response data in a database in chronological order, and self-correct based on the response learning data pairs.

[0315] Additionally, the server may acquire relationship data including at least some of the blood relationship data, friendship relationship data, and role relationship data between the agent and the user. It may further perform the steps of generating a relationship graph corresponding to the agent based on the relationship data, generating a relationship diagram based on at least some of the relationship graph and the rate of change of the relationship graph, and reflecting the relationship diagram in the emotion output data and risk level.

[0316] The server may change the status flag to post-mortem based on whether the user is alive, and when the status flag is changed to post-mortem, it may activate a pre-configured pre-mortem trigger. At this time, the pre-mortem trigger may include at least one of an anniversary, a calendar event, a geofence, and a connection of an operator terminal. It may further include a step of generating emotion output data when the pre-mortem trigger is detected. When the pre-configured pre-mortem trigger is detected, it may preemptively transmit appropriate emotion output data to the operator rather than reacting passively.

[0317] According to one embodiment, the digital self curation system (4010) is configured to manage datasets in a multi-layered hierarchical structure according to the source of creation, purpose of preservation, security level, and lifecycle of the data constituting the digital self. This data hierarchy is intended to simultaneously achieve service stability, ethics, and privacy protection, and some layers may be integrated or subdivided depending on the implementation environment and service policy.

[0318] First, Dataset 0 (P0) is meta-log data that includes the time of user interaction, dwell time, sentiment summary indicators, timeline summary information, etc., and consists of data in which individuals cannot be directly identified or the possibility of identification is minimized. Dataset 0 may be preserved for a long period or permanently for the purpose of maintaining the context of the service, improving quality, and statistical analysis, even after the original data has been deleted.

[0319] The first dataset (P1) includes original record data such as voice, video, text conversations, diaries, and biometric signals provided by the user during their lifetime or collected through the system, and is managed at the highest security level. The first dataset is utilized as core training data for digital self creation, but it is subject to permanent deletion via crypto-shredding according to the user's will or policy.

[0320] Dataset 1.5 (P1.5) is correction data to be prepared in case Dataset 1 is insufficient or missing, and is generated retrospectively through oral records, photo captions, memory inputs, etc. provided by authorized third parties such as family or acquaintances. In the event of an information conflict between Dataset 1 and Dataset 1.5, the system determines the priority according to a pre-set relationship grade or reliability criterion.

[0321] The third dataset (P3) is self-parameter data derived by training the first dataset or the 1.5 dataset, and includes the avatar's personality vector, speech model, response pattern, value weight, etc. The third dataset is stored logically separated from the original data and is maintained independently even after the original data is deleted, so that it is continuously utilized for generating responses of the digital self.

[0322] The fourth dataset (P4) is growth log data containing conversation history and event records accumulated during the process of interaction with the user or descendants after the digital self is activated, and is used as feedback data to update the third dataset in a time-series manner. Through this, the present invention can realize a growth-type digital self that changes and evolves according to interaction, rather than a static self fixed at the time of death.

[0323] As such, the digital self-curation system of the present invention provides a digital self lifecycle management structure that simultaneously satisfies personal information protection and service continuity through the separation of roles and organic linkage between data layers.

[0324] In this system, each dataset has the following organic flow. First, the first dataset (P1) is used as the basis (Seed) for learning and inference to form the third dataset (P3). However, once the third dataset (P3) is formed, even if the first dataset (P1) is deleted, the third dataset (P3) is independently stored and maintained and continuously used to generate responses from the digital self. Second, if the first dataset (P1) is insufficient or missing, the 1.5 dataset (P1.5) replaces or supplements it. In the event of a reliability conflict between the original (P1) and the corrected data (P1.5), the system determines the priority according to a pre-established relationship grade. Third, the fourth dataset (P4) is responsible for the evolution of the third dataset (P3). The fourth dataset functions as a feedback loop to realize a 'growth self' that changes through interaction, rather than a self that is fixed at the time of death.

[0325] Specific embodiments of the present invention have been described above.

[0326] However, those skilled in the art will understand that the spirit and scope of the present invention are not specifically limited to these particular embodiments, and that various modifications and variations are possible within the scope of not altering the essence of the invention.

[0327] Accordingly, the aforementioned embodiments are provided to fully inform those skilled in the art of the scope of the invention and should be understood as illustrative in all respects and not restrictive, and the invention is defined only by the scope of the claims.

Claims

1. Regarding digital self-curation platforms, A communication interface unit that communicates with a server; A receiving device for acquiring language input data including at least some of the operator's voice, handwritten notes, and text, and emotion input data including at least some of the operator's behavior, facial expressions, gaze, posture, gestures, tone of voice, intonation, and breathing; A self-generation device that generates emotion learning data based on the above emotion input data and generates emotion learning data pairs including emotion labeling data corresponding to the emotion learning data; An ethical judgment device that calculates the risk level of the emotional input data based on social ethical standards and community ethical standards, and refines or blocks the emotional input data according to the risk level; and A control unit that generates emotion output data including corresponding language output data and non-language output data based on the output results of the self-generation device and the ethics judgment device. A digital self-curation platform characterized by including 2. In Paragraph 1, The above-mentioned self-generating device is, A digital self-curation platform characterized by including a digital self-generation model that includes at least some of a pre-trained artificial intelligence, machine learning model, deep learning network, and statistical-based prediction model, which is trained based on self-generation basic data including at least some of a user's conversation history, diary, essay, memo, video, voice, and handwritten notes.

3. In Paragraph 1, The above-mentioned ethical judgment device is, A digital self-curation platform characterized by including an ethical judgment model that includes criteria for judging social norm risk and criteria for judging community norm risk corresponding to social norm taboo expressions and community taboo expressions.

4. In Paragraph 1, The above control unit is, A digital self-curation platform characterized by converting the above-mentioned emotion output data into response data including at least some of text, voice, avatar, light, video, hologram, and multimodal signals.

5. In Paragraph 4, A digital self-curation platform characterized by the above-mentioned response data being output as at least some of a digital monument, UI, display, holographic device, speaker, robot, VR device, AR device, and mobile device.

6. In Paragraph 1, The above control unit is, A digital self-curation platform characterized by providing the above-mentioned emotion output data as at least one of a family relationship-centered metaverse service, a family tree service, and a digital tombstone service.

7. In Paragraph 6, The above family tree service is, A digital self-curation platform characterized by automatically generating a family tree based on family member information and family relationship data from photos or videos.

8. In Paragraph 6, The above metaverse service is, A digital self-curation platform characterized by providing an avatar created based on family data to operate in an environment where a place of memory is recreated as a virtual space.

9. In Paragraph 1, The above-mentioned ethical judgment device is, A digital self-curation platform characterized by refining or blocking the emotion input data when the risk level of the emotion input data is above a preset standard.

10. In Paragraph 1, The above control unit is, A digital self-curation platform characterized by acquiring an agent's emotional response data corresponding to the above-mentioned emotional output data and reflecting it in self-correction learning.

11. In Paragraph 10, A digital self-curation platform characterized by the fact that the above-mentioned emotional response data includes at least some of verbal response data and non-verbal response data.

12. In Paragraph 1, The above control unit is, A digital self-curation platform characterized by storing the above-mentioned emotion output data and emotion response data on a server, including time information.

13. In Paragraph 1, The above control unit is, A digital self-curation platform characterized by changing the generation logic of emotion output data when the user status flag changes from living to deceased.

14. In Paragraph 13, The above control unit is, A digital self-curation platform characterized by utilizing at least one of an anniversary, a calendar event, a geofence, and a connection of an operator terminal as a preemptive firing trigger.

15. In Paragraph 1, The above control unit is, A digital self-curation platform characterized by generating emotion output data by reflecting the output results of the self-generation device and the ethics judgment device in relationship data and relationship graphs.

16. In Paragraph 1, The above server is, A digital self-curation platform characterized by storing original data as basic data for self-generation, and emotion learning data, emotion labeling data, and derived parameter data generated based on the original data separately.

17. In Paragraph 1, The above control unit is, A digital self curation platform characterized by maintaining the operation of a digital self generation model based on the derived parameter data even after the original data is deleted or converted to an inaccessible state.

18. In Paragraph 1, The above control unit is, A digital self-curation platform characterized by generating emotion output data by applying authorized correction data according to preset criteria when part of emotion learning data or emotion response data is missing.

19. In Paragraph 1, The above control unit is, A digital self-curation platform characterized by updating a digital self-generation model of the self-generation device based on a growth log including emotional output data and emotional response data.

20. Regarding digital self-curation methods, The server, A step of obtaining from a receiving device language input data including at least some of the operator's voice, handwriting, and text, and emotion input data including at least some of the operator's behavior, facial expression, gaze, posture, gesture, tone of voice, intonation, and breathing; A step of generating emotion learning data based on the above emotion input data, and generating emotion learning data pairs including emotion labeling data corresponding to the emotion learning data; A step of calculating the risk level of the emotion input data based on social ethics standards and community ethics standards, and refining or blocking the emotion input data if the risk level is greater than or equal to a preset standard risk level; and A step of generating emotion output data including corresponding language output data and non-language output data based on the output results of the self-generation device and the ethics judgment device; A digital self-curation method including 21. A computer-readable recording medium having a program for executing a digital self-curation method pursuant to paragraph 20.

22. In a digital self data lifecycle control device implemented by a server, A receiving unit that receives self-generation basic data including at least some of the user's voice, video, text conversation, diary, and biosignals from a user terminal through a communication interface unit; A self-generation device that generates emotion learning data, emotion labeling data, and self parameter data based on the above-mentioned self-generation basic data; A storage unit that logically separates and stores the above-mentioned self-generation basic data and the above-mentioned self-parameter data; and Includes a control unit, but The above control unit is, It is configured to generate emotional output data of the digital self based on the self parameter data even after the above-mentioned self generation basic data is deleted or permanently deleted, and If the above-mentioned basic data for self-generation is insufficient or missing, emotional output data is generated by applying correction data provided by at least one authorized third party, such as a family member or acquaintance. Characterized by being configured to update the self parameter data in a time series based on growth log data including emotional output data and emotional response data accumulated during the process of interaction with a user or operator after the digital self is activated. Digital self data life cycle control device.

23. In a method for controlling the digital self data lifecycle, The server, A step of receiving self-generation basic data from a user terminal, including at least some of the user's voice, video, text conversations, diaries, and biosignals; A step of generating emotion learning data, emotion labeling data, and self parameter data based on the above-mentioned self-generation basic data; A step of logically separating and storing the above-mentioned self-generation basic data and the above-mentioned self-parameter data; A step of generating emotional output data of a digital self based on the self parameter data even after the above-mentioned self generation basic data is deleted or permanently deleted; If the above-mentioned basic data for self-generation is insufficient or missing, a step of generating emotion output data by applying correction data provided by at least one authorized third party, such as a family member or acquaintance; and A step of updating the self-parameter data in a time-series manner based on growth log data including emotional output data and emotional response data accumulated during the process of interaction with a user or operator; A digital self data lifecycle control method including 24. A computer-readable recording medium having a program for executing the digital self data lifecycle method pursuant to paragraph 23.

Images