Following above, he keeps his Clansmen at a relative distance - close enough to care for them, but distant enough where he doesn't engage in a closer relationship with them. They're his Clansmen, he'd protect them, but at the same time, he doesn't want any of them to worry about him. It may make him seem somewhat cold, but it's the exact opposite; he just cares too much. This puts Seri in the same boat because Izumo doesn't want - or need - her friendship, not even a listening ear. She's a Blue, anyway, and Izumo's not too keen on getting close to them at all.