What part of 'unable to feel extreme emotions' do people not get? He can be vaguely attracted to people (we've seen it before in the anime) but he doesn't go all lustful and stuff anymore can he love anyone but his sister or get angry for himself (all those comments about her anger for his sake is nice because he can't feel it himself).
So no 'taking' Mayumi.
