This paper introduces a sentiment analysis method suitable to the human-agent and face-to-face interactions. We present the positioning of our system and its evaluation protocol according to the existing sentiment analysis literature and detail how the proposed system integrates the human-agent interaction issues. Finally, we provide an in-depth analysis of the results obtained by the evaluation, opening the discussion on the different difficulties and the remaining challenges of sentiment analysis in human-agent interactions.