With more people, there are more voices to tune out.
What Franklin Roosevelt did, which really offended them, was he strengthened the labor unions - made it possible for them to strike. The oligarchs were furious because the working class was not supposed to have any power at all.