838 results about "Interaction technology" patented technology

The invention relates to the technical field of building data interaction, in particular to a building engineering interaction method and system based on a BIM model and a medium. The method comprises the following steps: collecting construction site environment parameters and structure response data by using a distributed sensor network; preprocessing the data by an edge computing node; generating standardized environment data and key structure indexes; comparing environment threshold values to identify abnormal events; the method comprises the following steps of: displaying spatial positioning and risk levels through a visual interface, receiving a regulation and control instruction input by a user, dynamically adjusting construction parameters of a BIM model, generating an optimized construction progress scheme, comparing the optimized scheme with an original model to identify a construction conflict area, and generating a conflict resolution report. Through the advanced monitoring technology and construction management technology, the response speed and decision-making efficiency of the building engineering project are improved, and the safety guarantee and resource utilization efficiency in the construction process are improved.

The invention relates to the technical field of voice interaction, particularly discloses an AI intelligent robot voice interaction method and system, and aims to solve the problems of low voice interaction accuracy, insufficient reliability and lack of authority control in a complex noise environment. A dynamic noise feature library containing steady-state noise, impact noise and human voice interference features and a pre-stored gesture instruction library are constructed, audio signals are collected in real time, low-frequency-band, middle-frequency-band and high-frequency-band differential noise reduction is executed, Mel-frequency cepstral coefficient features are extracted, noise scenes are matched, corresponding voice recognition models are switched, and voice recognition is achieved. And calculating a confidence value of the voice instruction, outputting multi-modal verification data in combination with a dynamic confidence threshold, and outputting an authority control signal through voiceprint matching, authority verification and instruction consistency judgment. Through multi-modal fusion, dynamic adaptation and authority control, the voice recognition accuracy and interaction safety in a complex noise environment are remarkably improved, and the method is suitable for scenes such as factory intelligent inspection.

The invention discloses a database interaction method, device and equipment based on multi-agent cooperation and a medium, and relates to the technical field of database interaction. The method comprises the steps of obtaining a query request, performing semantic partitioning and vectorization, and obtaining a retrieval vector set; and selecting a search source and searching to obtain an information block. And obtaining candidate knowledge through knowledge base retrieval. And integrating the information blocks and the candidate knowledge to obtain a candidate information set. And constructing a dynamic demand understanding model according to the query request and the candidate information set, and obtaining a task state matrix and a tool calling task. And calling a corresponding auxiliary tool and operating, updating the task state matrix according to an operation result and returning the task state matrix to the task planning agent to update the tool calling task until the tool calling task is no longer output, and obtaining a task result. And when the database interaction module is called, generating a chart according to an output result of the database interaction module, and outputting the chart and a task result together. Otherwise, directly outputting a task result.

The invention, which relates to the technical field of intelligent interaction, discloses an AI-based adaptive scene intelligent interaction system comprising a multi-modal data acquisition module, a modal preprocessing module, a multi-modal embedded coding module, an intention fusion and representation module, a service scene matching module and a service execution and reinforcement learning module. The method comprises the following steps: acquiring multi-modal original data in a user interaction process, including voice signals, text input and user behavior tracks, and synchronously recording an acquisition timestamp; according to the method, through a multi-modal unified embedding and dynamic weighting mechanism, the problem of characteristic dimension imbalance is effectively solved, and the user intention recognition accuracy is improved; meanwhile, reinforcement learning and a multi-factor scoring model are combined, personalized scene matching and dynamic response are achieved, the adaptive capacity and service accuracy of the system in a complex environment are improved, and therefore the stability and user experience of the intelligent interaction system are remarkably optimized.

The invention discloses an intelligent interaction system based on a large language model and a knowledge graph, and relates to the technical field of artificial intelligence and intelligent interaction. According to the system, data processed by a data acquisition and preprocessing module is stored in a local knowledge warehouse in a three-layer nested structure; the multi-modal semantic understanding module constructs a knowledge system in which a triple knowledge graph and a vector database are complementary by optimizing a BERT model and performing dual-channel analysis; the intelligent interaction module realizes natural language interaction and business task automatic triggering based on an RAG technology and a dialogue memory mechanism; the information change identification and analysis module generates a personnel change risk report and performs early warning; and the summary report automatic generation module outputs a structured report. According to the method, the problems of data splitting, complex interaction and slow response in traditional enterprise management are solved, the functions of system intelligent question answering, personnel change risk analysis, structured report automatic generation and the like are realized, and the management efficiency and the information consistency are improved.

The invention discloses an Android system user interface interaction method and device based on a large language model, and relates to the technical field of intelligent man-machine interaction, and the method comprises the steps: receiving a natural language instruction of a user, and synchronously capturing a real-time interface state of a current Android screen to generate a structured context containing semantic information of interactive elements; inputting the instruction and the context into a large language model, and generating a hierarchical task plan formed by a plurality of atomic operations with a logic dependency relationship; based on a dynamic adaptation mechanism, converting the atomic operation sequence into an operation instruction which can be executed by the Android system; receiving interface change information after instruction execution, and feeding back the interface change information as a new round of context to the large language model to determine a task execution state; and finally generating natural language feedback information to the user according to the execution state. By means of the mode, complex cross-application tasks can be understood and executed, and interaction intelligence of the Android system is remarkably improved through real-time interface perception and dynamic planning adjustment.

The invention relates to the technical field of robot control and somatosensory interaction, and particularly discloses a knee joint exoskeleton robot control system based on somatosensory interaction. The system comprises a somatosensory signal acquisition and fusion module, a personalized dynamic model construction module, an adaptive intention decision module, a variable impedance compliance control module and a rehabilitation process evaluation and parameter self-tuning module. Through cooperative work of the modules, precise recognition of the motion intention of a wearer, online construction of an individualized biomechanical model and flexible and adaptive motion control are achieved, control parameters can be automatically adjusted according to the rehabilitation process, and therefore the flexibility and individuation level of man-machine interaction and the effectiveness of rehabilitation training are improved.

The invention relates to the technical field of man-machine interaction, in particular to a dialogue Agent interaction method based on multi-modal intention understanding, which comprises the following steps: S1, collecting multi-modal data in a user interaction process in real time, and calculating a time synchronization deviation value of each modal data source; s2, constructing a space-time fusion feature vector; s3, analyzing a dominant action instruction and a recessive behavior clue in the space-time fusion feature vector; s4, generating a multi-level intention analysis tree; s5, when the corrected confidence coefficient of any node in the intention analysis tree is lower than a set threshold value, activating a targeted sensor to complementarily collect data; and S6, analyzing a tree drive response decision according to the finally confirmed intention. According to the method, high-precision identification and response control of the dialogue Agent on the user intention in a complex scene are realized by constructing a multi-modal interaction method with space-time consistency fusion capability, an explicit and implicit intention analysis mechanism and an adaptive modal clarification strategy.

The invention provides a multi-intention processing method and device for a smart home, electronic equipment and a storage medium, and relates to the technical field of man-machine interaction. The smart home system can first judge whether intention conflicts exist among the different intentions, and if the intention conflicts exist, the smart home system can first calculate a first priority score corresponding to the intention map and a second priority score corresponding to the secondary intention; the method comprises the following steps: acquiring a first priority score and a second priority score, then performing strategy selection according to the first priority score and the second priority score, determining a conflict resolution strategy for the deliberate map and the secondary intention, and then executing a control instruction corresponding to the conflict resolution strategy, thereby improving intention understanding accuracy and response efficiency, and improving intention understanding experience. And the capability and decision of the system on complex intention conflicts are improved, the timeliness and flexibility of control instruction response are guaranteed, and the accuracy of system response is improved.

The invention relates to the technical field of man-machine interaction, and discloses an intelligent man-machine interaction system based on meta-universe, which is used for solving the problem of lagging caused by the fact that each virtual module possibly initiates response at the same time during man-machine interaction, and comprises the following steps: constructing a meta-universe scene, collecting multi-modal behavior information of a user in the meta-universe scene, and displaying the multi-modal behavior information of the user in the meta-universe scene. Performing unified formatting processing on the multi-modal behavior information to obtain formatted multi-modal behavior information, and performing joint reasoning on the formatted multi-modal behavior information by adopting a natural language processing engine and a multi-modal fusion model to obtain structured command data; the method comprises the following steps: obtaining a virtual module in a meta universe scene needing to make a response action according to structured command data, recording the virtual module as a driven module, obtaining priority information of all the driven modules, evaluating to obtain a priority index, and driving the modules to make the response action according to the priority index, thereby effectively improving the interaction response efficiency and the user experience continuity.

The invention discloses an e-commerce virtual scene generation method and system based on digital twinning and visual interaction, and belongs to the technical field of computer interaction. According to the method, the basic data set of the e-commerce scene is acquired, and then scene modeling is performed by using digital twinning, so that the initial virtual scene model containing the three-dimensional space layout of the commodity and the visual features of the environmental elements can be generated; performing visual interaction feature analysis on the initial model, and innovatively mining user interaction attention region features and interaction operation preference features as key bases for optimizing the virtual scene; and on the basis of the characteristics, interaction optimization adjustment is performed on the initial model, and finally the e-commerce virtual scene supporting dynamic visual interaction is generated, so that the user interaction experience of the e-commerce scene is comprehensively improved, the crossing from the traditional static display to the dynamic and intelligent interaction experience is realized, and the immersive interactive e-commerce virtual scene construction is realized.

The invention discloses a duplex-screen intelligent cabin interaction method and system, and relates to the technical field of intelligent cabin interaction, and the method comprises the steps: collecting user operation behavior data, vehicle state data and environment perception data in real time through a vehicle-mounted sensor group, a man-machine interaction interface and a vehicle bus; establishing a multi-dimensional user data model containing an interaction feature vector, a driving mode matrix and a preference configuration tree; the method comprises the following steps: analyzing user historical interaction data based on an adaptive weighting algorithm, and calculating use frequency weight coefficients of different functional modules; according to the duplex-screen intelligent cockpit interaction method and system provided by the invention, the precision of a user portrait is remarkably improved by constructing a multi-dimensional user data model, personalized interaction experience is realized, and the interface layout and interaction logic can be dynamically adjusted according to a real-time driving scene by adopting a self-adaptive weighting algorithm and a random forest classifier, so that the user experience is improved. The real-time performance and the adaptability of the system are enhanced, and the interaction experience is optimized.

The invention relates to the technical field of man-machine interaction, and discloses a multi-modal task interaction assisting method and system and computer equipment, and the method comprises the steps: collecting multi-modal interaction data of a user executing a multi-dimensional interaction task; determining a user behavior feature vector based on the multi-modal interaction data, and constructing a task completion quantitative model; determining the weight of each operation step by adopting a dynamic weight distribution strategy based on the multi-dimensional interaction task completion degree quantification model, and dynamically adjusting a preset adaptive coefficient by adopting a preset adjustment mechanism based on the multi-modal interaction data and the weight of each operation step; generating a dynamic task guiding strategy by adopting a dynamic path planning algorithm based on the weight of each operation step and a dynamically adjusted preset adaptive coefficient; and executing the dynamic task guiding strategy to obtain a multi-modal execution result. According to the method, a complete closed-loop process is formed from data acquisition, analysis, decision making to execution, intelligent services can be provided for middle-aged and elderly users, and the user experience and the task execution efficiency are improved.

The invention relates to the technical field of artificial intelligence interaction, in particular to a full-life-cycle AI accompanying agent system and method.The full-life-cycle AI accompanying agent system comprises a core engine layer, a business service layer, a basic tool layer, a data layer and an interface layer, and the full-life-cycle AI accompanying agent method comprises the steps that multi-modal data, including biological signals, behavior data, voice data, expression data and environment data, of a user are collected; constructing a user digital twinborn model, and updating model parameters through a weighted learning algorithm; identifying a user cognition development stage, and dynamically adjusting interaction complexity and an expression mode; emotional connection is established, and resonance feedback is generated through multi-mode emotional recognition and memory retrieval; according to the method, interaction memory is stored, cross-stage retrieval is realized, data management is performed by adopting a hierarchical memory architecture, and accurate judgment of a cognitive stage is realized by synchronously collecting multi-dimensional data such as behavior tracks, language texts and biological signals of a user and combining a development psychology theoretical model to construct a feature association map.

The invention discloses a generative AI model real-time rendering engine construction method and related equipment thereof, and relates to the technical field of man-machine interaction, and the method comprises the steps: obtaining multi-modal semantic information input by a user, and converting the multi-modal semantic information into a patent vector and a constraint parameter; constructing a lightweight generation model, and directly generating a renderable intermediate representation by adopting the lightweight generation model; constructing a rendering pipeline based on the generated intermediate representation, and converting the intermediate representation into a visual image; capturing user operation in real time, dynamically updating constraint conditions of a generated model, triggering increment generation, and optimizing system performance by adopting a model lightweight, hardware acceleration and hybrid rendering strategy; through the time filtering and space constraint technology, the generated content is consistent in time and space. According to the method, the problems of low generation speed and low rendering efficiency in the fusion process of the generative AI model and the real-time rendering engine are solved, and the efficiency and quality of visual image generation and rendering are improved.

The invention discloses a virtual interaction system for realizing multi-dimensional perception, which relates to the technical field of virtual interaction, and comprises a man-machine interaction unit, a mapping matching unit, a data processing unit and a data processing unit, and the coordinate matching module is responsible for establishing a coordinate matching relation of the virtual object in the virtual space according to the size, shape and attitude parameters of the virtual object. The interaction device is operated to control the intelligent mark point in the virtual interaction system to interact with the target object, according to the position relation in the interaction process, the position posture of the interaction object is collected to determine whether contact occurs, event triggering and target interaction are achieved through the triggering control unit, a user senses the event, and the interaction efficiency is improved. Interaction requirements can be met, multi-target interaction control requirements are met, and the purpose of saving system resources is achieved.

The embodiment of the invention relates to the technical field of man-machine interaction, and discloses a joint data feedback method and device based on a flexible sensor, a medium and a product. The method comprises the following steps: determining a time sequence calibration feature set according to a multi-source synchronization signal of a flexible sensor array; according to the time sequence calibration feature set, determining a joint function evaluation result and an action classification result; in combination with the joint function evaluation result, according to the deviation between the action classification result and a preset biomechanical standard library, joint dynamic function deviation data are determined; determining a biomechanical feedback instruction of the wearable device according to the joint dynamic function deviation data; wherein the biomechanical feedback instruction is matched with the functional state corresponding to the joint function evaluation result, and the method and device can be at least used for solving the technical problems that in the related technology, joint functions and actions are not accurate in recognition, and precise personalized control is difficult to achieve.

The invention discloses a multi-modal interaction evaluation system and method based on synchronous data acquisition, and relates to the technical field of biomedical engineering and man-machine interaction, and the method comprises the steps: generating a synchronous pulse sequence through a synchronous signal generator according to the same coding time base, and outputting the synchronous pulse sequence to an optical path and an electrical path at the same time; pulse events are recognized in the video and the electrophysiological data respectively, and observation pairs of counting numbers and timestamps are obtained; time drift is continuously estimated and compensated through robust regression in a sliding window, and a unified time axis is generated; furthermore, a dynamic time window of heart beat anchoring is constructed according to an R peak event of the electrocardiosignal, and consistent segmentation and feature extraction of cross-modal data are achieved.

The invention belongs to the technical field of vehicle safety control and man-machine interaction, and particularly relates to a man-machine co-driving takeover decision-making, stability control and optimization method, which comprises the following steps of: A, constructing a driving safety field model, reflecting the influence of a dynamic environment and road elements on a vehicle by constructing the safety field model, and providing a real-time basis for takeover decision-making; b, constructing a driver human factor capability field, and evaluating the takeover capability of the driver through multi-modal data acquisition and real-time analysis; c, dynamically coupling a safety field and a human factor capability field in the takeover decision; d, a man-machine takeover mode judgment and control power distribution method; e, the takeover scheme is screened and optimized, and further screening and adjustment are carried out through the comprehensive cost function of the optimization algorithm and used for being executed in actual operation. The method has the advantages that the handover between the automatic driving system and the driver is intelligently controlled by evaluating the environmental risk, the driver ability and the take-over time in real time, and it is ensured that driving right transfer is safely, timely and efficiently completed.

The invention relates to the technical field of artificial intelligence voice interaction, in particular to a semantic analysis and feedback method for voice interaction of intelligent navigation equipment. According to the method, through multi-dimensional semantic analysis and dynamic confidence evaluation, the ability of understanding the complex intention of the user in a noise environment is effectively improved, and loss of key information is avoided. Meanwhile, a dynamic content generation engine driven by the knowledge graph is utilized, personalized feedback and deep culture guide can be provided according to the intention of the user, and both basic information and cross-spatio-temporal knowledge nodes can be accurately presented. In addition, due to the introduction of an intelligent fault tolerance and intention learning mechanism, the man-machine cooperation efficiency is optimized, the interaction steps are reduced, the knowledge graph weight is dynamically adjusted through user behavior feedback, and the user satisfaction is remarkably improved. The innovative designs jointly bring smoother, more accurate and story-rich guide experience to the user.

The invention discloses a multi-mode induction display screen response method fusing voice interaction, and relates to the technical field of man-machine interaction, and the method comprises the following steps: in a user interaction process, calling a built-in sensor through an interface component real-time state monitoring mechanism, continuously monitoring a rendering process of a target interface component at a millisecond-level time granularity, and displaying the rendering process of the target interface component; collecting multi-dimensional availability state parameter information of the interface component; the method comprises the following steps: preprocessing collected interface component multi-dimensional availability state parameter information, and extracting key indexes reflecting potential asynchronous triggering risks from preprocessed data through a feature engineering method; by sensing the state of the interface component in real time, extracting the asynchronous risk index and combining machine learning recognition and TCN prediction modeling, dynamic regulation and control of the induction triggering window and the animation rhythm are achieved, the problem of mispointing or dislocation of the induction component is avoided, the accuracy and safety of interaction response are improved, and the user experience is improved. The method is suitable for high-precision scenes such as vehicle-mounted, medical and industrial control.

The invention relates to the technical field of artificial intelligence and man-machine interaction, and discloses a dynamic evaluation system and method for robustness of a graphical user interface agent, and the system comprises a task execution environment module, an abnormal triggering mechanism module, a state verification module, a data acquisition module and an evaluation index calculation module. The intelligent agent enters a simulation environment and completes a circulation interaction process of perception-decision-execution-feedback, abnormal events are monitored, automatically triggered, abnormal scene generation and recovery are carried out, state verification is carried out, whether tasks are successfully completed or not is judged, multi-source data acquisition of continuously recording the tasks in the whole operation period is carried out, and the task execution time is shortened. The method has the beneficial effects that the evaluation environment is converted from a static state to a dynamic state, the authenticity is obviously improved, a semantic-driven exception injection mechanism is introduced, context-aware interruption is realized, the evaluation is more objective, and the method has good expandability and reusability.

The invention discloses an information interaction optimization method and electronic equipment, and relates to the technical field of information interaction.The information interaction optimization method comprises the steps that when the first interaction duration of a substrate management controller is larger than or equal to a first interaction duration threshold value, the number of first interaction retry times is counted; when the number of times of first interaction retry is larger than or equal to a first retry number threshold value and the first interaction retry time average length corresponding to each retry is larger than or equal to a first interaction time length threshold value, the first interaction time length threshold value and the first retry number threshold value are adjusted, and a second interaction time length threshold value and a second interaction retry number threshold value are obtained; when the current interaction duration is greater than or equal to a second interaction duration threshold value, if the current interaction retry number is greater than or equal to a second interaction retry number threshold value, determining that interaction is abnormal; and triggering the substrate management controller and the target external equipment to execute an exception recovery mechanism. By shortening the waiting time and reducing the number of retry times, waste of resources is avoided, and the interaction efficiency is improved.

The invention relates to the technical field of digital robot intelligent interaction, in particular to an AI interaction method based on a digital robot, terminal equipment and a storage medium. Generating a space grid mask by using the depth map to segment a pixel region of the target object in the RGB image, when the shielding rate exceeds 30%, performing weighted calculation on a pixel displacement vector according to the depth map and the RGB image and generating a depth weighted optical flow field, constructing a target motion feature set in combination with a space-time gradient histogram descriptor and a feature point set, and inputting the target motion feature set into a space-time convolutional neural network to obtain a target motion feature set; a real track is output through the depth guidance three-dimensional convolution layer and the feature recombination module to drive the robot to move, the problems that in the traditional technology, the shielding scene track prediction deviation is large, and cross-modal feature fusion is insufficient are solved, and the track prediction precision and interaction reliability in a complex scene are improved.

The invention provides a voice dialogue interaction method and system for industrial equipment, and relates to the technical field of man-machine interaction, and the method comprises the following steps: obtaining a voice signal of a user, and carrying out the voice enhancement processing of the voice signal of the user through an incremental adaptive filtering algorithm, and obtaining an enhanced voice; according to the enhanced voice, using an information gain transfer learning method to identify an interaction intention of the user, and generating a to-be-interacted voice based on the interaction intention of the user; based on the enhanced voice, utilizing a time delay estimation method to identify an interaction position of the user, and performing position prediction on the position of the user; according to the position prediction result, an industrial equipment horn output strategy is constructed; and outputting the to-be-interacted voice based on the industrial equipment loudspeaker output strategy so as to realize voice dialogue interaction between the user and the industrial equipment. According to the invention, convenience, high efficiency and accuracy of interaction between the user and the equipment in an industrial scene are greatly improved, and intelligent development of industrial production is facilitated.

The invention discloses a gesture interaction method, an intelligent ring, a storage medium and a computer program product, and relates to the technical field of intelligent wearable equipment and human-computer interaction, and the gesture interaction method is applied to the intelligent ring. Under the condition that the pressing pressure reaches a preset pressure threshold value and the duration of the pressing pressure reaches a preset time threshold value, interaction data of the user and the intelligent ring are collected, and the interaction data comprise at least one of touch operation data and spatial movement data; and identifying a target operation gesture of the user according to the interaction data, and converting the target operation gesture into a control instruction and executing the control instruction. The technical problem of high false touch rate is solved.

The invention provides an intelligent examination and approval method, system and device based on a multi-modal large model and a medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: analyzing a multi-modal examination and approval material through a visual large model to obtain multi-modal feature information; performing semantic understanding by using the language large model to generate structured semantic data; performing entity matching, relationship verification and logical reasoning based on the knowledge graph to generate a research and judgment report; intelligent decision support and risk early warning are realized; approval instruction conversion, service evaluation and self-adaptive interaction are realized through a multi-mode interaction technology; and continuously optimizing the model and the knowledge base by using the approval process data. According to the invention, by constructing a technical framework of deep fusion of vision, language and knowledge, comprehensive intelligentization of the approval process is realized, and the beneficial effects of improving the approval efficiency, guaranteeing decision fairness, enhancing the complex material processing capability, improving the user experience and realizing continuous evolution of the system are achieved.

The invention relates to the technical field of artificial intelligence and man-machine interaction, and discloses an AI multi-mode everything identification child interaction star key system, which comprises a multi-mode perception module, a data storage module, a self-adaptive strategy decision module, a situation arrangement and retrieval module, a content generation and safety control module, a multi-mode expression execution module and an asynchronous capability evaluation module. The system maps visual features to a semantic space through an attention mechanism to analyze a multi-modal anaphora intention, determines a cognitive age stage based on a user dynamic portrait matrix and matches an interaction strategy, generates reply content by using a large language model, and dynamically adjusts voice and action output parameters according to a real-time emotion vector. And finally updating the portrait by using a moving average algorithm. The problems that in the prior art, visual semantic alignment is lacked, and the content difficulty cannot follow dynamic evolution of the user ability are solved, and accurate intention understanding and personalized growth interaction are achieved.

The invention discloses a human-computer interaction efficiency evaluation method for a central control screen of an intelligent automobile, and belongs to the technical field of intelligent automobile interaction. Determining use requirements of the central control screen; the method comprises the following steps: acquiring indexes in three aspects of driver response behavior, vehicle control and user experience, performing fuzzy C-means clustering on three dimensions for each vehicle using scene to obtain clustering scores of each sample on different dimensions in different vehicle using scenes, calculating weights of all dimensions in different vehicle using scenes according to the clustering scores, and calculating the weights of all the dimensions in different vehicle using scenes; calculating average clustering scores of all the samples in all the vehicle use scenes, and judging the man-machine interaction efficiency level of the central control screen of the intelligent vehicle; in combination with a Fuzzy C-means clustering algorithm and an entropy weight method, clustering results of different dimensions in each scene and the weight of each dimension in different vehicle use scenes are adaptively calculated, and the objectivity and generalization ability of data are improved.

The invention discloses a Rhatish culture VRAR virtual-real fusion interaction method based on eye movement tracking, and particularly relates to the technical field of virtual reality and augmented reality man-machine interaction. The method comprises the following steps: acquiring an eye image of a user through a near-infrared image acquisition unit, and inputting the eye image into a double-branch convolutional neural network model to estimate a two-dimensional fixation point of a screen; performing nonlinear correction on a fixation point based on the user personalized sight line offset correction model established in the offline calibration stage; in the online interaction stage, the corrected fixation point is combined with equipment pose data to be converted into three-dimensional space rays, the distance and angle relation between the three-dimensional space rays and an interaction target is calculated, and an interaction instruction is triggered when a threshold condition is continuously met. According to the method and the system, accurate, stable and natural eye movement interaction of the virtual contents of the Rhatish culture and the entity exhibits in the VR / AR environment is realized, and the user experience and the system robustness are improved.