The present invention relates to a system and method for implementing an assistive emotional companion for a user, wherein the system is designed for capturing emotional as well as performance feedback of a participant participating in an interactive session either with a system or with a presenter participant and utilizing such feedback to adaptively customize subsequent parts of the interactive session in an iterative manner. The interactive presentation can either be a live person talking and/or presenting in person, or a streaming video in an interactive chat session, and an interactive session can be a video gaming activity, an interactive simulation, an entertainment software, an adaptive education training system, or the like. The physiological responses measured will be a combination of facial expression analysis, and voice expression analysis. Optionally, other signals such as camera based heart rate and/or touch based skin conductance may be included in certain embodiments.