I believe you felt you had every reason to.

And I am not exactly the most gentlest of men. There are occasions, I think, one should yell at me. I think myself the unmovable mountain. Genji calls it stubbornness.

[It's spoken wryly. In any case, it caused him to take her seriously and he was not insulted by it.]

My approach to this matter with Koltira aside, you do not quite seem yourself. I would listen to you, if you wish to speak your concerns to me.