This talk explores what happens when AI systems are designed not only to respond to our emotions but to influence them. Emotional AI tools, including chatbots and AI companions, increasingly use affective design, reward loops and adaptive emotional cues that can quietly shape user behavior. These techniques, grounded in behavioral psychology and neuroeconomics, raise new questions about autonomy, manipulation and the limits of traditional consent.
Ana Catarina de Alencar shows how emotionally responsive AI can trigger patterns of dependency, create subtle forms of influence and blur the boundary between persuasion and manipulation. When an AI system comforts, validates, or rewards users at precisely the right emotional moment, can we still say that consent is freely given? And what does informed consent mean when the system itself affects the user’s emotional state and decision-making?
Drawing from emerging real-world cases (including situations of emotional attachment to AI companions), the talk highlights why existing privacy and consent frameworks are insufficient for emotionally interactive technologies. Attendees will gain a clear understanding of how manipulative design manifests in Emotional AI, why it matters for ethics and regulation, and what needs to change to protect user autonomy.
Ana Catarina de Alencar is an international lawyer and ethicist based in Paris, working at the intersection of AI governance, regulation, and philosophy. She is the Resident Philosopher at The AI Collective, a San Francisco–based organization dedicated to shaping responsible AI futures. Ana is also a PhD researcher at Université de Lille, where she develops an interdisciplinary project on emotional AI, bridging law, philosophy, and neurobiology. She holds a Master’s degree in Philosophy and Technology of Law and is the author of several publications, including Artificial Intelligence, Ethics and Law (2022).
