People who are against hate are not a fringe minority, not even a silent majority, but are a silenced majority, silenced by the corporate media.
I think the more people you add on stage, the more locked it things become. Whereas fewer members on stage gets the audience to notice any small idiosyncrasy or unique moment that one of the three people onstage is having. It also means that mistakes are accentuated, but to good affect generally if you are that type of band.